[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 03:41:19.457005 31273 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.138.126:41693
I20260806 03:41:19.459944 31273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:41:19.461606 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:19.492777 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:19.494072 31283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:19.492815 31282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:20.882845 31284 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260806 03:41:20.882989 31273 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260806 03:41:20.888309 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:20.888623 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:20.888835 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987680888804 us; error 0 us; skew 500 ppm
I20260806 03:41:20.890041 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:20.897867 31273 webserver.cc:460] Webserver started at http://127.30.138.126:33097/ using document root <none> and password file <none>
I20260806 03:41:20.899288 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:20.899549 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:20.900204 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:20.909024 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/master-0-root/instance:
uuid: "788391b0417d4284b4ccf9781bfdb3b5"
format_stamp: "Formatted at 2026-08-06 03:41:20 on dist-test-slave-tk9z"
I20260806 03:41:20.921031 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260806 03:41:20.928340 31293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:20.930125 31273 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260806 03:41:20.930682 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/master-0-root
uuid: "788391b0417d4284b4ccf9781bfdb3b5"
format_stamp: "Formatted at 2026-08-06 03:41:20 on dist-test-slave-tk9z"
I20260806 03:41:20.931102 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:20.974164 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:20.976187 31273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:41:20.976770 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:21.063053 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.126:41693
I20260806 03:41:21.063149 31369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.126:41693 every 8 connection(s)
I20260806 03:41:21.075037 31370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:21.104926 31370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: Bootstrap starting.
I20260806 03:41:21.113044 31370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:21.115674 31370 log.cc:824] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: Log is configured to *not* fsync() on all Append() calls
I20260806 03:41:21.121551 31370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: No bootstrap required, opened a new log
I20260806 03:41:21.147490 31370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER }
I20260806 03:41:21.148272 31370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:21.148612 31370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788391b0417d4284b4ccf9781bfdb3b5, State: Initialized, Role: FOLLOWER
I20260806 03:41:21.149998 31370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER }
I20260806 03:41:21.150751 31370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:41:21.151085 31370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:41:21.151464 31370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:21.155938 31370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER }
I20260806 03:41:21.157059 31370 leader_election.cc:302] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788391b0417d4284b4ccf9781bfdb3b5; no voters:
I20260806 03:41:21.159271 31370 leader_election.cc:288] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:41:21.159641 31374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:21.163403 31374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 1 LEADER]: Becoming Leader. State: Replica: 788391b0417d4284b4ccf9781bfdb3b5, State: Running, Role: LEADER
I20260806 03:41:21.164621 31374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER }
I20260806 03:41:21.165859 31370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:41:21.180048 31376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788391b0417d4284b4ccf9781bfdb3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER } }
I20260806 03:41:21.180850 31376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:41:21.182271 31375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788391b0417d4284b4ccf9781bfdb3b5. Latest consensus state: current_term: 1 leader_uuid: "788391b0417d4284b4ccf9781bfdb3b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788391b0417d4284b4ccf9781bfdb3b5" member_type: VOTER } }
I20260806 03:41:21.183058 31375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [sys.catalog]: This master's current role is: LEADER
I20260806 03:41:21.184424 31390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:41:21.198064 31390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:41:21.200343 31273 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:41:21.218626 31390 catalog_manager.cc:1347] Generated new cluster ID: f681190e4ce74e088abfbdb68e4c9953
I20260806 03:41:21.218950 31390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:41:21.240248 31390 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:41:21.242230 31390 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:41:21.258867 31390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: Generated new TSK 0
I20260806 03:41:21.260089 31390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:41:21.268455 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:21.280812 31403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.283001 31404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.287752 31406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:21.287914 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:21.288764 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:21.288990 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:21.289168 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987681289136 us; error 0 us; skew 500 ppm
I20260806 03:41:21.289783 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:21.294080 31273 webserver.cc:460] Webserver started at http://127.30.138.65:37713/ using document root <none> and password file <none>
I20260806 03:41:21.294708 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:21.294946 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:21.295238 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:21.296572 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-0-root/instance:
uuid: "f2aaabfd88234360b4fb0550085cac61"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.302670 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 03:41:21.307337 31411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.308274 31273 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 03:41:21.308625 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
uuid: "f2aaabfd88234360b4fb0550085cac61"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.308955 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:21.330641 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:21.332037 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:21.335330 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:21.340180 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:21.340502 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.340816 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:21.340972 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.452523 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.65:45451
I20260806 03:41:21.452677 31502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.65:45451 every 8 connection(s)
I20260806 03:41:21.460556 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:21.472306 31511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.486402 31515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.488148 31517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:21.489022 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:21.489917 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:21.490141 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:21.490350 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987681490332 us; error 0 us; skew 500 ppm
I20260806 03:41:21.491062 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:21.496639 31504 heartbeater.cc:347] Connected to a master server at 127.30.138.126:41693
I20260806 03:41:21.496874 31273 webserver.cc:460] Webserver started at http://127.30.138.66:36775/ using document root <none> and password file <none>
I20260806 03:41:21.497159 31504 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:21.497586 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:21.497892 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:21.498267 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:21.498499 31504 heartbeater.cc:511] Master 127.30.138.126:41693 requested a full tablet report, sending...
I20260806 03:41:21.500041 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-1-root/instance:
uuid: "0a69965c26904db58123d8b355f03ae2"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.503294 31320 ts_manager.cc:194] Registered new tserver with Master: f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451)
I20260806 03:41:21.506685 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:41:21.507863 31320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51904
I20260806 03:41:21.514844 31524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.515743 31273 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260806 03:41:21.516048 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
uuid: "0a69965c26904db58123d8b355f03ae2"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.516306 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:21.539839 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:21.541503 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:21.544641 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:21.548003 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:21.548202 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.548534 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:21.548677 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.667289 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.66:39407
I20260806 03:41:21.667656 31623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.66:39407 every 8 connection(s)
I20260806 03:41:21.681311 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:21.690052 31628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.695035 31630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:21.697546 31632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:21.700284 31624 heartbeater.cc:347] Connected to a master server at 127.30.138.126:41693
I20260806 03:41:21.700776 31624 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:21.701606 31624 heartbeater.cc:511] Master 127.30.138.126:41693 requested a full tablet report, sending...
I20260806 03:41:21.703636 31320 ts_manager.cc:194] Registered new tserver with Master: 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:21.704824 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:21.705691 31320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51912
I20260806 03:41:21.705991 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:21.706239 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:21.706476 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987681706458 us; error 0 us; skew 500 ppm
I20260806 03:41:21.707233 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:21.710598 31273 webserver.cc:460] Webserver started at http://127.30.138.67:45317/ using document root <none> and password file <none>
I20260806 03:41:21.711113 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:21.711342 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:21.711609 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:21.712705 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-2-root/instance:
uuid: "a6e18afe5269433c825d5b598e2302e0"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.717821 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 03:41:21.721580 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.722534 31273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:41:21.722869 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
uuid: "a6e18afe5269433c825d5b598e2302e0"
format_stamp: "Formatted at 2026-08-06 03:41:21 on dist-test-slave-tk9z"
I20260806 03:41:21.723205 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785987679226243-31273-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:21.748062 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:21.749329 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:21.751690 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:21.755470 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:21.755712 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.755952 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:21.756122 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:21.857489 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.67:38529
I20260806 03:41:21.858070 31728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.67:38529 every 8 connection(s)
I20260806 03:41:21.877882 31729 heartbeater.cc:347] Connected to a master server at 127.30.138.126:41693
I20260806 03:41:21.878278 31729 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:21.879122 31729 heartbeater.cc:511] Master 127.30.138.126:41693 requested a full tablet report, sending...
I20260806 03:41:21.881319 31320 ts_manager.cc:194] Registered new tserver with Master: a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:21.882854 31320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51916
I20260806 03:41:21.883265 31273 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018576724s
I20260806 03:41:21.887605 31273 test_util.cc:268] Using random seed: -1453709990
I20260806 03:41:21.948520 31320 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51926:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:41:21.953442 31320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:41:22.097872 31451 tablet_service.cc:1467] Processing CreateTablet for tablet bb63622f25704c2e8c76cc0a5b577812 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:41:22.097954 31452 tablet_service.cc:1467] Processing CreateTablet for tablet f8925cebe2dc42fc8cacb7f8094fc8ae (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:41:22.099388 31446 tablet_service.cc:1467] Processing CreateTablet for tablet ff71bdd5817f4e58ab2f9f040e93bfd9 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:41:22.097940 31449 tablet_service.cc:1467] Processing CreateTablet for tablet 04868dda0d8343c9976e9c058425b828 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:41:22.101846 31451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb63622f25704c2e8c76cc0a5b577812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.107574 31446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff71bdd5817f4e58ab2f9f040e93bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.108776 31440 tablet_service.cc:1467] Processing CreateTablet for tablet 36108fe9044b420da02f5d5065c34021 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:41:22.113142 31447 tablet_service.cc:1467] Processing CreateTablet for tablet b23aa35b40d44524b86a004bb1122b20 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:41:22.115724 31445 tablet_service.cc:1467] Processing CreateTablet for tablet 1881a43bbca74cfba2311ba0e7fe2d0f (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:41:22.107695 31442 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76ec03e104b158aac90e1fb81bbb3 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:41:22.116814 31449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04868dda0d8343c9976e9c058425b828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.122298 31673 tablet_service.cc:1467] Processing CreateTablet for tablet 04868dda0d8343c9976e9c058425b828 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:41:22.127662 31440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36108fe9044b420da02f5d5065c34021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.126539 31676 tablet_service.cc:1467] Processing CreateTablet for tablet f8925cebe2dc42fc8cacb7f8094fc8ae (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:41:22.122550 31444 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9cab51a0d48fbaf69e141590d7556 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:41:22.135038 31670 tablet_service.cc:1467] Processing CreateTablet for tablet ff71bdd5817f4e58ab2f9f040e93bfd9 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:41:22.133494 31675 tablet_service.cc:1467] Processing CreateTablet for tablet bb63622f25704c2e8c76cc0a5b577812 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:41:22.131990 31671 tablet_service.cc:1467] Processing CreateTablet for tablet b23aa35b40d44524b86a004bb1122b20 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:41:22.144414 31445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881a43bbca74cfba2311ba0e7fe2d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.134074 31568 tablet_service.cc:1467] Processing CreateTablet for tablet f8925cebe2dc42fc8cacb7f8094fc8ae (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:41:22.148240 31563 tablet_service.cc:1467] Processing CreateTablet for tablet b23aa35b40d44524b86a004bb1122b20 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:41:22.150583 31673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04868dda0d8343c9976e9c058425b828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.136058 31567 tablet_service.cc:1467] Processing CreateTablet for tablet bb63622f25704c2e8c76cc0a5b577812 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:41:22.152647 31669 tablet_service.cc:1467] Processing CreateTablet for tablet 1881a43bbca74cfba2311ba0e7fe2d0f (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:41:22.166725 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 04868dda0d8343c9976e9c058425b828 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:41:22.146229 31562 tablet_service.cc:1467] Processing CreateTablet for tablet ff71bdd5817f4e58ab2f9f040e93bfd9 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:41:22.167696 31668 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9cab51a0d48fbaf69e141590d7556 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:41:22.171324 31444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9cab51a0d48fbaf69e141590d7556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.172513 31567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb63622f25704c2e8c76cc0a5b577812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.172997 31675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb63622f25704c2e8c76cc0a5b577812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.178609 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04868dda0d8343c9976e9c058425b828. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.181579 31668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9cab51a0d48fbaf69e141590d7556. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.183326 31667 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76ec03e104b158aac90e1fb81bbb3 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:41:22.182314 31561 tablet_service.cc:1467] Processing CreateTablet for tablet 1881a43bbca74cfba2311ba0e7fe2d0f (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:41:22.189680 31560 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9cab51a0d48fbaf69e141590d7556 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:41:22.190711 31447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23aa35b40d44524b86a004bb1122b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.192752 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff71bdd5817f4e58ab2f9f040e93bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.196522 31670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff71bdd5817f4e58ab2f9f040e93bfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.198566 31559 tablet_service.cc:1467] Processing CreateTablet for tablet 38c76ec03e104b158aac90e1fb81bbb3 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:41:22.202610 31663 tablet_service.cc:1467] Processing CreateTablet for tablet 36108fe9044b420da02f5d5065c34021 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:41:22.207823 31558 tablet_service.cc:1467] Processing CreateTablet for tablet 36108fe9044b420da02f5d5065c34021 (DEFAULT_TABLE table=test-workload [id=6bbea680fca0404e9d9b784118cfd061]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:41:22.208299 31560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9cab51a0d48fbaf69e141590d7556. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:41:22.213202 31388 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:41:22.214838 31568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8925cebe2dc42fc8cacb7f8094fc8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.215291 31669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881a43bbca74cfba2311ba0e7fe2d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.216507 31558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36108fe9044b420da02f5d5065c34021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.220016 31452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8925cebe2dc42fc8cacb7f8094fc8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.225680 31676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8925cebe2dc42fc8cacb7f8094fc8ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.227579 31563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23aa35b40d44524b86a004bb1122b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.229818 31561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1881a43bbca74cfba2311ba0e7fe2d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.232982 31663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36108fe9044b420da02f5d5065c34021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.232679 31442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76ec03e104b158aac90e1fb81bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.234725 31559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76ec03e104b158aac90e1fb81bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.245766 31671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23aa35b40d44524b86a004bb1122b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.258050 31667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c76ec03e104b158aac90e1fb81bbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:22.401641 31761 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.453032 31765 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:22.487056 31761 tablet_bootstrap.cc:654] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.489311 31764 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.516259 31504 heartbeater.cc:503] Master 127.30.138.126:41693 was elected leader, sending a full tablet report...
I20260806 03:41:22.532294 31765 tablet_bootstrap.cc:654] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.535717 31764 tablet_bootstrap.cc:654] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.537297 31761 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:22.541179 31761 ts_tablet_manager.cc:1397] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.141s user 0.014s sys 0.012s
I20260806 03:41:22.539417 31765 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:22.542847 31765 ts_tablet_manager.cc:1397] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.008s
I20260806 03:41:22.545655 31765 raft_consensus.cc:348] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.545712 31761 raft_consensus.cc:348] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.546794 31765 raft_consensus.cc:374] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.546857 31761 raft_consensus.cc:374] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.547372 31765 raft_consensus.cc:729] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.547379 31761 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.548820 31765 consensus_queue.cc:260] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.548822 31761 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.555895 31729 heartbeater.cc:503] Master 127.30.138.126:41693 was elected leader, sending a full tablet report...
I20260806 03:41:22.562347 31765 ts_tablet_manager.cc:1428] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.019s user 0.009s sys 0.003s
I20260806 03:41:22.562999 31761 ts_tablet_manager.cc:1428] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.021s user 0.006s sys 0.014s
I20260806 03:41:22.565970 31765 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:22.565991 31761 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.564666 31764 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.571609 31764 ts_tablet_manager.cc:1397] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.021s
I20260806 03:41:22.581048 31764 raft_consensus.cc:348] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.581966 31764 raft_consensus.cc:374] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.582309 31764 raft_consensus.cc:729] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.583189 31764 consensus_queue.cc:260] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.584478 31761 tablet_bootstrap.cc:654] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.585215 31765 tablet_bootstrap.cc:654] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.601524 31764 ts_tablet_manager.cc:1428] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.030s user 0.009s sys 0.008s
I20260806 03:41:22.602582 31624 heartbeater.cc:503] Master 127.30.138.126:41693 was elected leader, sending a full tablet report...
I20260806 03:41:22.608347 31764 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.609490 31761 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:22.610004 31761 ts_tablet_manager.cc:1397] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.005s
I20260806 03:41:22.613124 31761 raft_consensus.cc:348] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.613933 31761 raft_consensus.cc:374] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.614323 31765 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:22.614858 31765 ts_tablet_manager.cc:1397] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.018s
I20260806 03:41:22.614283 31761 raft_consensus.cc:729] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.615319 31764 tablet_bootstrap.cc:654] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
W20260806 03:41:22.616451 31731 tablet.cc:2369] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:41:22.615866 31761 consensus_queue.cc:260] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.618299 31765 raft_consensus.cc:348] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.619158 31765 raft_consensus.cc:374] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.619496 31765 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.619706 31761 ts_tablet_manager.cc:1428] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
I20260806 03:41:22.620468 31765 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.623200 31765 ts_tablet_manager.cc:1428] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260806 03:41:22.624259 31765 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:22.621076 31761 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.628122 31764 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.628674 31764 ts_tablet_manager.cc:1397] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20260806 03:41:22.631091 31761 tablet_bootstrap.cc:654] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.631088 31765 tablet_bootstrap.cc:654] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.631431 31764 raft_consensus.cc:348] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.632247 31764 raft_consensus.cc:374] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.632740 31764 raft_consensus.cc:729] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.633782 31764 consensus_queue.cc:260] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.636013 31764 ts_tablet_manager.cc:1428] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260806 03:41:22.636873 31764 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.637854 31770 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.638509 31770 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.643967 31764 tablet_bootstrap.cc:654] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.653254 31770 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:22.654506 31761 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:22.655047 31764 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.655210 31761 ts_tablet_manager.cc:1397] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.000s
I20260806 03:41:22.655546 31764 ts_tablet_manager.cc:1397] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.003s
I20260806 03:41:22.658054 31761 raft_consensus.cc:348] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.658077 31764 raft_consensus.cc:348] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.658931 31761 raft_consensus.cc:374] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.659057 31764 raft_consensus.cc:374] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.659260 31761 raft_consensus.cc:729] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.659499 31764 raft_consensus.cc:729] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.660013 31761 consensus_queue.cc:260] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.660476 31764 consensus_queue.cc:260] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.663132 31764 ts_tablet_manager.cc:1428] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260806 03:41:22.664213 31764 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.670581 31761 ts_tablet_manager.cc:1428] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260806 03:41:22.671649 31761 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.671962 31764 tablet_bootstrap.cc:654] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.684829 31764 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.685501 31764 ts_tablet_manager.cc:1397] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.000s
I20260806 03:41:22.688951 31764 raft_consensus.cc:348] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.690155 31764 raft_consensus.cc:374] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.690686 31764 raft_consensus.cc:729] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.691771 31761 tablet_bootstrap.cc:654] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.692193 31764 consensus_queue.cc:260] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.695478 31764 ts_tablet_manager.cc:1428] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 03:41:22.696903 31764 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.710981 31764 tablet_bootstrap.cc:654] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.730623 31761 tablet_bootstrap.cc:492] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:22.731585 31761 ts_tablet_manager.cc:1397] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.060s user 0.018s sys 0.000s
I20260806 03:41:22.736799 31765 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:22.737401 31765 ts_tablet_manager.cc:1397] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.113s user 0.035s sys 0.063s
I20260806 03:41:22.741482 31761 raft_consensus.cc:348] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.742260 31761 raft_consensus.cc:374] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.742620 31761 raft_consensus.cc:729] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.743506 31761 consensus_queue.cc:260] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.748144 31764 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.747355 31765 raft_consensus.cc:348] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.748675 31764 ts_tablet_manager.cc:1397] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.052s user 0.023s sys 0.001s
I20260806 03:41:22.749879 31761 ts_tablet_manager.cc:1428] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 03:41:22.754077 31761 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.760926 31761 tablet_bootstrap.cc:654] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.761564 31468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.750058 31765 raft_consensus.cc:374] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.762646 31468 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 0.
I20260806 03:41:22.763177 31764 raft_consensus.cc:348] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.763870 31764 raft_consensus.cc:374] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.764165 31764 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.764395 31644 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:22.764921 31764 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.776116 31761 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:22.762760 31765 raft_consensus.cc:729] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.783236 31764 ts_tablet_manager.cc:1428] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.032s user 0.011s sys 0.006s
I20260806 03:41:22.788754 31761 ts_tablet_manager.cc:1397] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.002s
I20260806 03:41:22.768107 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:22.789915 31765 consensus_queue.cc:260] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.765386 31770 raft_consensus.cc:2781] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:22.792392 31770 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:22.792758 31770 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.793852 31761 raft_consensus.cc:348] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.795003 31761 raft_consensus.cc:374] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.795392 31761 raft_consensus.cc:729] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.799134 31764 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:22.801513 31770 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.796214 31761 consensus_queue.cc:260] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.804453 31765 ts_tablet_manager.cc:1428] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.067s user 0.009s sys 0.002s
I20260806 03:41:22.808734 31765 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:22.808980 31793 raft_consensus.cc:482] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.809458 31793 raft_consensus.cc:504] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.811733 31793 leader_election.cc:288] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
W20260806 03:41:22.812024 31645 leader_election.cc:341] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.812214 31793 raft_consensus.cc:482] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.812608 31793 raft_consensus.cc:504] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.813812 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:22.814330 31793 leader_election.cc:288] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:22.814471 31587 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 0.
I20260806 03:41:22.816461 31468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.816809 31645 leader_election.cc:302] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:22.817102 31468 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 0.
I20260806 03:41:22.817813 31794 raft_consensus.cc:2781] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:22.818222 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.818642 31773 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.818938 31466 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 0.
I20260806 03:41:22.819075 31773 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.820992 31644 leader_election.cc:302] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:22.821496 31773 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:22.807045 31764 tablet_bootstrap.cc:654] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.822312 31792 raft_consensus.cc:2781] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:22.816474 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:22.823912 31792 raft_consensus.cc:482] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:22.824985 31792 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.818220 31794 raft_consensus.cc:482] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:41:22.825933 31645 leader_election.cc:341] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.826061 31794 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.831894 31773 raft_consensus.cc:482] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.832602 31770 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:22.832399 31773 raft_consensus.cc:504] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.833647 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
W20260806 03:41:22.835278 31645 leader_election.cc:341] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.835541 31773 leader_election.cc:288] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:22.834163 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:22.836359 31466 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.840425 31765 tablet_bootstrap.cc:654] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.844327 31761 ts_tablet_manager.cc:1428] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.055s user 0.008s sys 0.000s
I20260806 03:41:22.839633 31794 raft_consensus.cc:504] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.867887 31466 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 1.
I20260806 03:41:22.869557 31644 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters: 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:22.870738 31770 raft_consensus.cc:2781] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:22.875965 31770 raft_consensus.cc:686] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:22.877580 31770 consensus_queue.cc:237] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.883368 31794 leader_election.cc:288] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:22.884958 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:22.886096 31587 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.889793 31765 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:22.890355 31765 ts_tablet_manager.cc:1397] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.011s
I20260806 03:41:22.882840 31761 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:22.893289 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:22.891649 31792 raft_consensus.cc:504] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.898427 31466 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.899415 31765 raft_consensus.cc:348] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.900381 31765 raft_consensus.cc:374] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.901168 31765 raft_consensus.cc:729] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.902201 31765 consensus_queue.cc:260] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.906888 31765 ts_tablet_manager.cc:1428] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20260806 03:41:22.908423 31773 raft_consensus.cc:482] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.909286 31773 raft_consensus.cc:504] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.909524 31765 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:22.911705 31773 leader_election.cc:288] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:22.916595 31587 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 1.
I20260806 03:41:22.922053 31645 leader_election.cc:302] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:22.928970 31773 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:22.929582 31773 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.926472 31770 raft_consensus.cc:2781] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:22.944929 31466 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 1.
I20260806 03:41:22.945840 31792 leader_election.cc:288] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:22.950188 31764 tablet_bootstrap.cc:492] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:22.950215 31773 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:22.951252 31770 raft_consensus.cc:686] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:22.951260 31764 ts_tablet_manager.cc:1397] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.162s user 0.026s sys 0.018s
I20260806 03:41:22.955473 31468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.957593 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:22.955624 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:22.955492 31770 consensus_queue.cc:237] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.959095 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.958189 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.959251 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:22.959959 31761 tablet_bootstrap.cc:654] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
W20260806 03:41:22.961023 31645 leader_election.cc:341] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:41:22.963580 31529 leader_election.cc:341] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.963286 31463 raft_consensus.cc:2454] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 0.
I20260806 03:41:22.951712 31316 catalog_manager.cc:5495] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
W20260806 03:41:22.964654 31529 leader_election.cc:341] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.960194 31466 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.964896 31764 raft_consensus.cc:348] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
W20260806 03:41:22.965510 31529 leader_election.cc:341] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:22.965845 31764 raft_consensus.cc:374] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:22.966620 31764 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:22.966892 31529 leader_election.cc:302] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:22.967315 31764 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.967849 31773 raft_consensus.cc:2781] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:22.968851 31773 raft_consensus.cc:482] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:22.970068 31773 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:22.974892 31765 tablet_bootstrap.cc:654] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:22.980623 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:22.988157 31773 raft_consensus.cc:504] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:22.991010 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:22.994760 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.000350 31694 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 0.
I20260806 03:41:23.003010 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.003919 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:23.004483 31463 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.003320 31773 leader_election.cc:288] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.006235 31773 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.007047 31773 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.009053 31773 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
W20260806 03:41:23.010740 31529 leader_election.cc:341] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.014725 31529 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2; no voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61
W20260806 03:41:23.016920 31529 leader_election.cc:341] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.018738 31529 leader_election.cc:302] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters: f2aaabfd88234360b4fb0550085cac61
W20260806 03:41:23.020116 31529 leader_election.cc:341] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.020563 31529 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2; no voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.022204 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:23.023317 31464 raft_consensus.cc:2379] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a69965c26904db58123d8b355f03ae2 in current term 1: Already voted for candidate a6e18afe5269433c825d5b598e2302e0 in this term.
I20260806 03:41:23.027979 31463 raft_consensus.cc:2454] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 1.
I20260806 03:41:23.035033 31773 raft_consensus.cc:2726] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.047758 31529 leader_election.cc:302] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.048449 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.049293 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.047160 31773 raft_consensus.cc:2726] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:41:23.051007 31529 leader_election.cc:341] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.051308 31761 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:23.053283 31771 raft_consensus.cc:482] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.053748 31805 raft_consensus.cc:2781] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:23.054278 31805 raft_consensus.cc:482] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:23.053941 31771 raft_consensus.cc:504] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.051918 31761 ts_tablet_manager.cc:1397] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.175s user 0.026s sys 0.037s
I20260806 03:41:23.056203 31773 raft_consensus.cc:2781] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.058061 31773 raft_consensus.cc:686] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:23.057821 31529 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2; no voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.061980 31761 raft_consensus.cc:348] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.063021 31761 raft_consensus.cc:374] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.063408 31761 raft_consensus.cc:729] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.064783 31761 consensus_queue.cc:260] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.066685 31773 consensus_queue.cc:237] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.071085 31771 leader_election.cc:288] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.054709 31805 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.083103 31764 ts_tablet_manager.cc:1428] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.129s user 0.020s sys 0.036s
I20260806 03:41:23.079635 31466 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 1.
I20260806 03:41:23.084316 31764 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:23.086615 31644 leader_election.cc:302] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters: 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:23.087581 31810 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.091747 31770 raft_consensus.cc:2781] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.092865 31770 raft_consensus.cc:686] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:23.095880 31761 ts_tablet_manager.cc:1428] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.037s user 0.010s sys 0.014s
I20260806 03:41:23.097054 31761 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:23.094295 31770 consensus_queue.cc:237] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.126852 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
W20260806 03:41:23.128908 31414 leader_election.cc:341] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.135033 31810 raft_consensus.cc:2726] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.182040 31771 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.182806 31771 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.189925 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.190524 31771 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.195961 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.197351 31583 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 0.
I20260806 03:41:23.198202 31765 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:23.196245 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.198709 31765 ts_tablet_manager.cc:1397] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.289s user 0.015s sys 0.008s
I20260806 03:41:23.192342 31805 raft_consensus.cc:504] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
W20260806 03:41:23.199743 31414 leader_election.cc:341] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:41:23.199172 31415 leader_election.cc:341] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.204788 31765 raft_consensus.cc:348] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.203251 31415 leader_election.cc:302] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2aaabfd88234360b4fb0550085cac61; no voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:23.205785 31761 tablet_bootstrap.cc:654] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.206023 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.206916 31694 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.205528 31765 raft_consensus.cc:374] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.207705 31771 raft_consensus.cc:2726] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.209928 31765 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.207701 31805 leader_election.cc:288] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.213744 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:23.213311 31764 tablet_bootstrap.cc:654] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.214268 31765 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.215399 31415 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters: a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:23.219583 31771 raft_consensus.cc:2781] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:23.220144 31771 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:23.220548 31771 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:41:23.223461 31529 leader_election.cc:341] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.225764 31694 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 1.
I20260806 03:41:23.226325 31765 ts_tablet_manager.cc:1428] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.027s user 0.009s sys 0.000s
I20260806 03:41:23.229252 31765 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:23.229255 31529 leader_election.cc:302] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters: f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.230624 31805 raft_consensus.cc:2781] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.231102 31805 raft_consensus.cc:686] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:23.231917 31805 consensus_queue.cc:237] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.232352 31771 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.236372 31764 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:23.236703 31761 tablet_bootstrap.cc:492] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:23.237110 31764 ts_tablet_manager.cc:1397] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.153s user 0.031s sys 0.036s
I20260806 03:41:23.237229 31761 ts_tablet_manager.cc:1397] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.140s user 0.018s sys 0.028s
I20260806 03:41:23.239851 31771 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.240777 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:23.240777 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.241483 31583 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:41:23.242763 31414 leader_election.cc:341] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.242640 31764 raft_consensus.cc:348] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.243443 31764 raft_consensus.cc:374] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.243844 31764 raft_consensus.cc:729] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.244594 31764 consensus_queue.cc:260] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.251057 31583 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 1.
I20260806 03:41:23.249670 31761 raft_consensus.cc:348] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.252449 31765 tablet_bootstrap.cc:654] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.252079 31761 raft_consensus.cc:374] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.253568 31761 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.254014 31415 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters: a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:23.254855 31764 ts_tablet_manager.cc:1428] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260806 03:41:23.255463 31771 raft_consensus.cc:2781] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.255267 31761 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.256090 31764 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:23.255932 31771 raft_consensus.cc:686] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:23.258878 31811 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.257874 31771 consensus_queue.cc:237] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.260353 31761 ts_tablet_manager.cc:1428] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260806 03:41:23.260169 31811 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.261780 31761 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:23.262712 31811 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.264487 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.266782 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.275617 31583 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 0.
I20260806 03:41:23.275616 31694 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 0.
I20260806 03:41:23.276933 31415 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.277746 31811 raft_consensus.cc:2781] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:23.278208 31811 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:23.269168 31764 tablet_bootstrap.cc:654] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.278609 31811 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.286844 31761 tablet_bootstrap.cc:654] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.288695 31765 tablet_bootstrap.cc:492] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:23.289351 31765 ts_tablet_manager.cc:1397] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.012s
I20260806 03:41:23.290172 31770 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.290148 31811 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.294693 31765 raft_consensus.cc:348] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.295470 31765 raft_consensus.cc:374] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.295586 31764 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:23.295778 31765 raft_consensus.cc:729] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.296043 31764 ts_tablet_manager.cc:1397] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.009s
I20260806 03:41:23.295723 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.296478 31694 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.296721 31765 consensus_queue.cc:260] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.298148 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:23.298854 31587 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.301057 31764 raft_consensus.cc:348] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.301754 31764 raft_consensus.cc:374] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.302029 31764 raft_consensus.cc:729] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.304000 31764 consensus_queue.cc:260] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.306016 31587 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 1.
I20260806 03:41:23.306492 31811 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.307294 31415 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.308521 31694 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 1.
I20260806 03:41:23.310976 31764 ts_tablet_manager.cc:1428] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260806 03:41:23.311106 31765 ts_tablet_manager.cc:1428] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.021s user 0.006s sys 0.002s
W20260806 03:41:23.311214 31388 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:41:23.311918 31764 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: Bootstrap starting.
I20260806 03:41:23.311918 31765 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:23.315938 31811 raft_consensus.cc:2781] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.316632 31811 raft_consensus.cc:686] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:23.317575 31811 consensus_queue.cc:237] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.319689 31764 tablet_bootstrap.cc:654] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.321030 31765 tablet_bootstrap.cc:654] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.328042 31319 catalog_manager.cc:5495] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:23.341638 31319 catalog_manager.cc:5495] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:23.376770 31761 tablet_bootstrap.cc:492] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:23.378952 31761 ts_tablet_manager.cc:1397] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.117s user 0.018s sys 0.015s
I20260806 03:41:23.384896 31761 raft_consensus.cc:348] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.385560 31764 tablet_bootstrap.cc:492] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: No bootstrap required, opened a new log
I20260806 03:41:23.385713 31761 raft_consensus.cc:374] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.386188 31761 raft_consensus.cc:729] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.386087 31764 ts_tablet_manager.cc:1397] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: Time spent bootstrapping tablet: real 0.074s user 0.029s sys 0.028s
I20260806 03:41:23.387415 31794 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260806 03:41:23.395116 31764 raft_consensus.cc:348] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.387523 31770 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20260806 03:41:23.420415 31764 raft_consensus.cc:374] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.420759 31764 raft_consensus.cc:729] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.407548 31761 consensus_queue.cc:260] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.423673 31761 ts_tablet_manager.cc:1428] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.042s user 0.010s sys 0.002s
I20260806 03:41:23.424717 31761 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: Bootstrap starting.
I20260806 03:41:23.430452 31761 tablet_bootstrap.cc:654] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.421548 31764 consensus_queue.cc:260] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.411408 31318 catalog_manager.cc:5495] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.432367 31764 ts_tablet_manager.cc:1428] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: Time spent starting tablet: real 0.045s user 0.013s sys 0.002s
I20260806 03:41:23.447098 31318 catalog_manager.cc:5495] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.411937 31320 catalog_manager.cc:5495] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.452384 31320 catalog_manager.cc:5495] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.474581 31770 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.488943 31805 consensus_queue.cc:1035] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.507990 31765 tablet_bootstrap.cc:492] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:23.508507 31765 ts_tablet_manager.cc:1397] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.197s user 0.038s sys 0.037s
I20260806 03:41:23.521734 31792 raft_consensus.cc:482] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.522400 31792 raft_consensus.cc:504] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.524938 31792 leader_election.cc:288] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:23.526408 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.531353 31805 raft_consensus.cc:482] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.532084 31805 raft_consensus.cc:504] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.533159 31765 raft_consensus.cc:348] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.536824 31805 leader_election.cc:288] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.536710 31765 raft_consensus.cc:374] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.537704 31765 raft_consensus.cc:729] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.538694 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:23.541963 31765 consensus_queue.cc:260] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.543972 31644 leader_election.cc:302] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0; no voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.544754 31792 raft_consensus.cc:2726] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.544020 31462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:23.545539 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
W20260806 03:41:23.546778 31529 leader_election.cc:341] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:41:23.549640 31529 leader_election.cc:341] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:23.550057 31529 leader_election.cc:302] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2; no voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.551225 31808 raft_consensus.cc:2726] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.552985 31765 ts_tablet_manager.cc:1428] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.044s user 0.009s sys 0.000s
I20260806 03:41:23.554584 31694 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.555754 31765 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:23.555904 31761 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: No bootstrap required, opened a new log
I20260806 03:41:23.556574 31761 ts_tablet_manager.cc:1397] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: Time spent bootstrapping tablet: real 0.132s user 0.045s sys 0.057s
I20260806 03:41:23.558908 31761 raft_consensus.cc:348] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.559700 31761 raft_consensus.cc:374] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.560045 31761 raft_consensus.cc:729] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.560703 31761 consensus_queue.cc:260] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.563109 31761 ts_tablet_manager.cc:1428] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 03:41:23.564900 31773 consensus_queue.cc:1035] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.578274 31765 tablet_bootstrap.cc:654] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.594775 31765 tablet_bootstrap.cc:492] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:23.595394 31765 ts_tablet_manager.cc:1397] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.007s
I20260806 03:41:23.598122 31765 raft_consensus.cc:348] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.598845 31765 raft_consensus.cc:374] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.599108 31765 raft_consensus.cc:729] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.599694 31765 consensus_queue.cc:260] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.601639 31765 ts_tablet_manager.cc:1428] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 03:41:23.602773 31765 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: Bootstrap starting.
I20260806 03:41:23.604264 31792 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.608145 31765 tablet_bootstrap.cc:654] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:23.615130 31773 raft_consensus.cc:482] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.615602 31773 raft_consensus.cc:504] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.618158 31773 leader_election.cc:288] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.619801 31765 tablet_bootstrap.cc:492] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: No bootstrap required, opened a new log
I20260806 03:41:23.620766 31765 ts_tablet_manager.cc:1397] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260806 03:41:23.621222 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.622089 31694 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 0.
I20260806 03:41:23.623219 31765 raft_consensus.cc:348] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.623961 31765 raft_consensus.cc:374] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:23.623739 31529 leader_election.cc:302] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:23.624689 31765 raft_consensus.cc:729] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Initialized, Role: FOLLOWER
I20260806 03:41:23.625346 31765 consensus_queue.cc:260] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.629055 31583 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.629213 31765 ts_tablet_manager.cc:1428] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:41:23.625283 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:23.635800 31464 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 0.
I20260806 03:41:23.635279 31773 raft_consensus.cc:2781] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:23.636785 31792 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.637529 31773 raft_consensus.cc:482] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:23.638101 31773 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.649317 31773 raft_consensus.cc:504] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.651976 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:23.652663 31464 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.654663 31771 raft_consensus.cc:482] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.655146 31771 raft_consensus.cc:504] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.657124 31771 leader_election.cc:288] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.657830 31773 leader_election.cc:288] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.658514 31694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.659224 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.659755 31694 raft_consensus.cc:2379] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2aaabfd88234360b4fb0550085cac61 in current term 1: Already voted for candidate 0a69965c26904db58123d8b355f03ae2 in this term.
I20260806 03:41:23.659839 31690 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.664450 31464 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 1.
I20260806 03:41:23.665619 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.665772 31529 leader_election.cc:302] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.666867 31773 raft_consensus.cc:2781] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.667434 31415 leader_election.cc:302] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2aaabfd88234360b4fb0550085cac61; no voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:23.667793 31773 raft_consensus.cc:686] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:23.668191 31690 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 1.
I20260806 03:41:23.669162 31771 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.668959 31773 consensus_queue.cc:237] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.678084 31771 raft_consensus.cc:2726] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.678761 31318 catalog_manager.cc:5495] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.687785 31792 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.688207 31792 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.689883 31792 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:23.690899 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2aaabfd88234360b4fb0550085cac61" is_pre_election: true
I20260806 03:41:23.691162 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.691954 31587 raft_consensus.cc:2379] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e18afe5269433c825d5b598e2302e0 in current term 1: Already voted for candidate f2aaabfd88234360b4fb0550085cac61 in this term.
I20260806 03:41:23.692798 31645 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0; no voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.693521 31792 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.698485 31792 raft_consensus.cc:2726] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:23.725804 31771 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.731483 31773 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.748855 31810 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:23.764397 31771 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.778402 31872 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.798442 31875 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:23.801887 31866 raft_consensus.cc:482] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:23.802443 31866 raft_consensus.cc:504] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.804510 31866 leader_election.cc:288] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.806200 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0" is_pre_election: true
I20260806 03:41:23.806360 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2" is_pre_election: true
I20260806 03:41:23.806876 31688 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 0.
I20260806 03:41:23.807065 31583 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 0.
I20260806 03:41:23.807729 31414 leader_election.cc:302] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.808799 31866 raft_consensus.cc:2781] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:23.809219 31866 raft_consensus.cc:482] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:23.809659 31866 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.817147 31866 raft_consensus.cc:504] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.820258 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:23.820554 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:23.820899 31583 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.821165 31688 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:23.819478 31866 leader_election.cc:288] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:23.826778 31688 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 1.
I20260806 03:41:23.828537 31414 leader_election.cc:302] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:23.828799 31583 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 1.
I20260806 03:41:23.830750 31866 raft_consensus.cc:2781] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:23.831323 31866 raft_consensus.cc:686] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:23.832229 31866 consensus_queue.cc:237] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:23.842794 31319 catalog_manager.cc:5495] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:23.949090 31583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04868dda0d8343c9976e9c058425b828"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:55412
I20260806 03:41:23.949746 31583 raft_consensus.cc:595] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Received request to transfer leadership to TS f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:23.968901 31688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:43922
I20260806 03:41:23.969650 31688 raft_consensus.cc:595] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:23.975656 31688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:43922
I20260806 03:41:23.976266 31688 raft_consensus.cc:595] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:23.981153 31688 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:43922
I20260806 03:41:23.981739 31688 raft_consensus.cc:595] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:23.989734 31583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36108fe9044b420da02f5d5065c34021"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:55412
I20260806 03:41:23.990253 31583 raft_consensus.cc:595] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Received request to transfer leadership to TS a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:23.993865 31808 raft_consensus.cc:979] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:23.994443 31773 raft_consensus.cc:1067] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Signalling peer a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:23.996338 31688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36108fe9044b420da02f5d5065c34021"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:43916
I20260806 03:41:23.997022 31688 raft_consensus.cc:482] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:23.997454 31688 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.004154 31688 raft_consensus.cc:504] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.006186 31688 leader_election.cc:288] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:24.007045 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:24.007340 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36108fe9044b420da02f5d5065c34021" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:24.007632 31464 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.008205 31583 raft_consensus.cc:3032] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:24.008562 31583 raft_consensus.cc:729] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:24.009330 31583 consensus_queue.cc:260] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.010599 31583 raft_consensus.cc:3037] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.014088 31464 raft_consensus.cc:2454] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:24.015446 31583 raft_consensus.cc:2454] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:24.015282 31644 leader_election.cc:302] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:24.016197 31792 raft_consensus.cc:2781] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:24.016671 31792 raft_consensus.cc:686] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:24.017539 31792 consensus_queue.cc:237] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.023701 31319 catalog_manager.cc:5495] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 1 to 2, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:24.048825 31841 raft_consensus.cc:979] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.049315 31792 raft_consensus.cc:1067] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.051160 31583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55394
I20260806 03:41:24.052642 31583 raft_consensus.cc:482] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:24.053040 31583 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.059564 31583 raft_consensus.cc:504] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.061502 31583 leader_election.cc:288] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:24.062958 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:24.062247 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:24.063773 31464 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.063774 31688 raft_consensus.cc:3032] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:24.064415 31688 raft_consensus.cc:729] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:24.065313 31688 consensus_queue.cc:260] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.069046 31841 raft_consensus.cc:979] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.069466 31688 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.069540 31792 raft_consensus.cc:1067] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.071368 31583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55394
I20260806 03:41:24.072026 31583 raft_consensus.cc:482] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:24.072274 31464 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.072466 31583 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.073560 31529 leader_election.cc:302] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:24.074474 31773 raft_consensus.cc:2781] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:24.074930 31773 raft_consensus.cc:686] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:24.075736 31773 consensus_queue.cc:237] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.079620 31583 raft_consensus.cc:504] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.080229 31688 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.081583 31583 leader_election.cc:288] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:24.083534 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:24.084143 31463 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.084710 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:24.085407 31688 raft_consensus.cc:3032] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:24.085794 31688 raft_consensus.cc:729] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:24.085651 31319 catalog_manager.cc:5495] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 1 to 2, leader changed from a6e18afe5269433c825d5b598e2302e0 (127.30.138.67) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:24.086786 31688 consensus_queue.cc:260] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.088174 31688 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.092164 31463 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.093315 31529 leader_election.cc:302] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:24.094694 31773 raft_consensus.cc:2781] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:24.094972 31688 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.095201 31773 raft_consensus.cc:686] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:24.096040 31773 consensus_queue.cc:237] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.103225 31319 catalog_manager.cc:5495] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 1 to 2, leader changed from a6e18afe5269433c825d5b598e2302e0 (127.30.138.67) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:24.123832 31773 consensus_queue.cc:1035] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:24.135727 31773 consensus_queue.cc:1035] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:24.350518 31773 raft_consensus.cc:979] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:24.351032 31808 raft_consensus.cc:1067] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Signalling peer f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:24.352671 31463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04868dda0d8343c9976e9c058425b828"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:46768
I20260806 03:41:24.353238 31463 raft_consensus.cc:482] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:24.353613 31463 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.360879 31463 raft_consensus.cc:504] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.363020 31463 leader_election.cc:288] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 2 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:24.363981 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:24.363998 31870 consensus_queue.cc:1035] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:24.364753 31583 raft_consensus.cc:3032] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:24.364817 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:24.365203 31583 raft_consensus.cc:729] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:24.365479 31688 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.366153 31583 consensus_queue.cc:260] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.370626 31583 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.372848 31688 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 2.
I20260806 03:41:24.374392 31414 leader_election.cc:302] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:24.375578 31870 raft_consensus.cc:2781] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:24.376441 31870 raft_consensus.cc:686] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:24.377475 31870 consensus_queue.cc:237] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.381074 31583 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 2.
I20260806 03:41:24.387198 31870 consensus_queue.cc:1035] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:24.391188 31318 catalog_manager.cc:5495] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 1 to 2, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 2 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:24.433697 31841 raft_consensus.cc:979] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.434242 31792 raft_consensus.cc:1067] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:24.436029 31583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55394
I20260806 03:41:24.436730 31583 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:24.437151 31583 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.444257 31583 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.446301 31583 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:24.447391 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:24.447599 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:24.448060 31463 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.448127 31690 raft_consensus.cc:3032] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:24.448637 31690 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:24.449453 31690 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.451164 31690 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:24.455425 31463 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.456535 31529 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:24.457391 31808 raft_consensus.cc:2781] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:24.457626 31690 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:24.457871 31808 raft_consensus.cc:686] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:24.458807 31808 consensus_queue.cc:237] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:24.467190 31319 catalog_manager.cc:5495] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 1 to 2, leader changed from a6e18afe5269433c825d5b598e2302e0 (127.30.138.67) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:24.472333 31463 raft_consensus.cc:1261] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.473948 31792 consensus_queue.cc:1035] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.483093 31583 raft_consensus.cc:1261] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.485217 31841 consensus_queue.cc:1035] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:24.539774 31690 raft_consensus.cc:1261] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.540989 31808 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.545469 31688 raft_consensus.cc:1261] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.546931 31773 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.554036 31463 raft_consensus.cc:1261] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.555546 31808 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.567729 31463 raft_consensus.cc:1261] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.569911 31810 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.811971 31583 raft_consensus.cc:1261] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.813261 31870 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:24.822137 31690 raft_consensus.cc:1261] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.823783 31875 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:24.991968 31690 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:24.993196 31810 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:25.006441 31463 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:25.008066 31890 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:27.020326 31273 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 03:41:27.021286 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:27.050469 31463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
mode: GRACEFUL
new_leader_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:46770
I20260806 03:41:27.051024 31463 raft_consensus.cc:595] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Received request to transfer leadership to TS a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:27.063478 31583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:27.064006 31583 raft_consensus.cc:595] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Received request to transfer leadership to TS a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:27.065096 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:41:27.065579 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:27.427454 31773 raft_consensus.cc:979] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:27.427918 31911 raft_consensus.cc:1067] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Signalling peer a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:27.429332 31688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:43916
I20260806 03:41:27.429781 31688 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:27.430033 31688 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:27.434413 31688 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.436038 31688 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 3 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:27.436931 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:27.437203 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:27.437451 31463 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:27.437738 31583 raft_consensus.cc:3032] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:27.438025 31583 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:27.438730 31583 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.439643 31583 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:27.442128 31463 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 3.
I20260806 03:41:27.443261 31644 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:27.444293 31901 raft_consensus.cc:2781] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:27.444450 31583 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 3.
I20260806 03:41:27.446511 31901 raft_consensus.cc:686] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:27.447465 31901 consensus_queue.cc:237] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.455363 31319 catalog_manager.cc:5495] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 2 to 3, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 3 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:27.487421 31905 raft_consensus.cc:979] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: : Instructing follower a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:27.487874 31905 raft_consensus.cc:1067] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Signalling peer a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:27.489346 31688 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:43918
I20260806 03:41:27.489886 31688 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:27.490214 31688 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:27.494895 31688 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.496539 31688 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:27.497584 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:27.497608 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:27.498215 31463 raft_consensus.cc:3032] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:27.498291 31583 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:27.498585 31463 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:27.499444 31463 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.500612 31463 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:27.504082 31583 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:27.505071 31645 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:27.505867 31463 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:27.505911 31901 raft_consensus.cc:2781] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:27.506486 31901 raft_consensus.cc:686] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:27.507375 31901 consensus_queue.cc:237] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:27.515316 31319 catalog_manager.cc:5495] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 1 to 2, leader changed from f2aaabfd88234360b4fb0550085cac61 (127.30.138.65) to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:27.911317 31463 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:27.912971 31901 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:27.923903 31583 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:27.926460 31913 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:28.026695 31583 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:28.028002 31913 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:28.039276 31463 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:28.041152 31931 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:28.066957 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:28.072796 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:28.073309 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:29.074191 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:29.081193 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:29.081710 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:30.083051 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:30.089387 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:30.089859 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:31.090772 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:31.095567 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:31.095978 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:32.096809 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:32.101790 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:32.102200 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:33.102973 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:33.107445 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:33.107823 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:34.108630 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:34.113708 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:34.114089 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:35.114948 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:35.121359 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:35.121827 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:36.122769 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:36.128417 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:36.128757 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:37.156993 31273 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 03:41:37.173676 31463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:48100
I20260806 03:41:37.174536 31463 raft_consensus.cc:595] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:37.181360 31463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:48100
I20260806 03:41:37.181924 31463 raft_consensus.cc:595] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:37.187462 31464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04868dda0d8343c9976e9c058425b828"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:48100
I20260806 03:41:37.188133 31464 raft_consensus.cc:595] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:37.193378 31690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:43922
I20260806 03:41:37.193933 31690 raft_consensus.cc:595] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:37.199976 31690 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
mode: GRACEFUL
new_leader_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:43922
I20260806 03:41:37.200668 31690 raft_consensus.cc:595] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Received request to transfer leadership to TS 0a69965c26904db58123d8b355f03ae2
I20260806 03:41:37.234970 31955 raft_consensus.cc:979] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.235543 31958 raft_consensus.cc:1067] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.237166 31587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55394
I20260806 03:41:37.237632 31587 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:37.237931 31587 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.243491 31587 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.245213 31587 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 3 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:37.246318 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:37.246513 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:37.247087 31464 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.247159 31690 raft_consensus.cc:3032] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:37.247524 31690 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:37.248183 31690 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.249197 31690 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.253408 31464 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 3.
I20260806 03:41:37.254714 31529 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:37.255415 31690 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 3.
I20260806 03:41:37.255811 31943 raft_consensus.cc:2781] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:37.256930 31943 raft_consensus.cc:686] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:37.257784 31943 consensus_queue.cc:237] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.265489 31318 catalog_manager.cc:5495] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 2 to 3, leader changed from a6e18afe5269433c825d5b598e2302e0 (127.30.138.67) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:37.335004 31951 raft_consensus.cc:979] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.335445 31951 raft_consensus.cc:1067] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.336975 31587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55406
I20260806 03:41:37.337512 31587 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:37.337772 31587 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.342470 31587 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.344444 31587 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:37.345373 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:37.345386 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:37.346101 31690 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.346168 31464 raft_consensus.cc:3032] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:37.346542 31464 raft_consensus.cc:729] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:37.347225 31464 consensus_queue.cc:260] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.348218 31464 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.352293 31690 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:37.353610 31529 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:37.353834 31464 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:37.354485 31943 raft_consensus.cc:2781] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:37.354993 31943 raft_consensus.cc:686] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:37.355077 31967 raft_consensus.cc:979] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.355657 31967 raft_consensus.cc:1067] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.355966 31943 consensus_queue.cc:237] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.357420 31587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04868dda0d8343c9976e9c058425b828"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55406
I20260806 03:41:37.357965 31587 raft_consensus.cc:482] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:37.358263 31587 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.364166 31587 raft_consensus.cc:504] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.364496 31318 catalog_manager.cc:5495] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 1 to 2, leader changed from f2aaabfd88234360b4fb0550085cac61 (127.30.138.65) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:37.366086 31587 leader_election.cc:288] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 3 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:37.366904 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:37.367202 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04868dda0d8343c9976e9c058425b828" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:37.367486 31464 raft_consensus.cc:3032] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:37.367856 31464 raft_consensus.cc:729] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:37.367945 31690 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.368639 31464 consensus_queue.cc:260] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.369794 31464 raft_consensus.cc:3037] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:37.374272 31690 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 3.
I20260806 03:41:37.375391 31529 leader_election.cc:302] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:37.376070 31464 raft_consensus.cc:2454] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 3.
I20260806 03:41:37.376102 31943 raft_consensus.cc:2781] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:37.376823 31943 raft_consensus.cc:686] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:37.377638 31943 consensus_queue.cc:237] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.385210 31318 catalog_manager.cc:5495] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 2 to 3, leader changed from f2aaabfd88234360b4fb0550085cac61 (127.30.138.65) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:37.572279 31955 raft_consensus.cc:979] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.572785 31958 raft_consensus.cc:1067] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.574134 31587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55394
I20260806 03:41:37.574543 31587 raft_consensus.cc:482] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:37.574754 31587 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:37.578496 31587 raft_consensus.cc:504] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.579828 31587 leader_election.cc:288] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 4 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:37.580685 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:37.580842 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff71bdd5817f4e58ab2f9f040e93bfd9" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:37.581301 31464 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:37.581380 31690 raft_consensus.cc:3032] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:41:37.581727 31690 raft_consensus.cc:729] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:37.582337 31690 consensus_queue.cc:260] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.583302 31690 raft_consensus.cc:3037] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:37.585971 31464 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 4.
I20260806 03:41:37.586941 31529 leader_election.cc:302] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:37.587667 31943 raft_consensus.cc:2781] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:41:37.588019 31690 raft_consensus.cc:2454] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 4.
I20260806 03:41:37.588266 31943 raft_consensus.cc:686] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 4 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:37.589205 31943 consensus_queue.cc:237] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.595765 31318 catalog_manager.cc:5495] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 3 to 4, leader changed from a6e18afe5269433c825d5b598e2302e0 (127.30.138.67) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:37.698585 31690 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:37.699843 31943 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:37.710220 31464 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:37.711742 31967 raft_consensus.cc:979] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: : Instructing follower 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.712208 31967 raft_consensus.cc:1067] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Signalling peer 0a69965c26904db58123d8b355f03ae2 to start an election
I20260806 03:41:37.712900 31959 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:41:37.713815 31587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
from {username='slave'} at 127.0.0.1:55406
I20260806 03:41:37.714447 31587 raft_consensus.cc:482] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:37.714835 31587 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.723145 31587 raft_consensus.cc:504] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.725059 31587 leader_election.cc:288] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:37.726100 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:37.726042 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8925cebe2dc42fc8cacb7f8094fc8ae" candidate_uuid: "0a69965c26904db58123d8b355f03ae2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:37.726738 31463 raft_consensus.cc:3032] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:37.726832 31690 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.727646 31463 raft_consensus.cc:729] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:37.728437 31463 consensus_queue.cc:260] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.729959 31463 raft_consensus.cc:3037] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:37.733371 31690 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:37.734321 31529 leader_election.cc:302] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, a6e18afe5269433c825d5b598e2302e0; no voters:
I20260806 03:41:37.735080 31943 raft_consensus.cc:2781] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:37.735510 31943 raft_consensus.cc:686] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:37.735942 31463 raft_consensus.cc:2454] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a69965c26904db58123d8b355f03ae2 in term 2.
I20260806 03:41:37.736330 31943 consensus_queue.cc:237] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:37.742887 31319 catalog_manager.cc:5495] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 reported cstate change: term changed from 1 to 2, leader changed from f2aaabfd88234360b4fb0550085cac61 (127.30.138.65) to 0a69965c26904db58123d8b355f03ae2 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "0a69965c26904db58123d8b355f03ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:37.867479 31690 raft_consensus.cc:1261] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:37.868589 31943 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:37.877576 31463 raft_consensus.cc:1261] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:37.880234 31943 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:37.916766 31690 raft_consensus.cc:1261] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:37.918001 31943 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:37.927924 31463 raft_consensus.cc:1261] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:37.929765 31943 consensus_queue.cc:1035] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:37.981595 31463 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:41:37.983011 31981 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:41:37.995517 31690 raft_consensus.cc:1261] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:41:37.997071 31943 consensus_queue.cc:1035] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:41:38.130205 31690 raft_consensus.cc:1261] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:38.131673 31943 consensus_queue.cc:1035] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:38.140427 31463 raft_consensus.cc:1261] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer 0a69965c26904db58123d8b355f03ae2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:38.141885 31981 consensus_queue.cc:1035] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:40.241408 31273 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 03:41:40.242136 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:40.249379 31587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:40.250245 31587 raft_consensus.cc:595] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Received request to transfer leadership to TS a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:40.252847 31587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:40.253295 31587 raft_consensus.cc:595] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Received request to transfer leadership to TS a6e18afe5269433c825d5b598e2302e0
I20260806 03:41:40.255564 31587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:40.255935 31587 raft_consensus.cc:595] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Received request to transfer leadership to TS f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:40.258016 31587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:40.258452 31587 raft_consensus.cc:595] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Received request to transfer leadership to TS f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:40.260530 31587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812"
dest_uuid: "0a69965c26904db58123d8b355f03ae2"
mode: GRACEFUL
new_leader_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:55414
I20260806 03:41:40.260982 31587 raft_consensus.cc:595] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Received request to transfer leadership to TS f2aaabfd88234360b4fb0550085cac61
I20260806 03:41:40.261754 31990 raft_consensus.cc:979] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.262037 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:41:40.262323 31959 raft_consensus.cc:1067] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Signalling peer f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.262579 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:40.263876 31463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:46768
I20260806 03:41:40.264384 31463 raft_consensus.cc:482] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:40.264703 31463 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.270501 31463 raft_consensus.cc:504] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.272639 31463 leader_election.cc:288] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 3 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:40.273928 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:40.274180 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9cab51a0d48fbaf69e141590d7556" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:40.275194 31690 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.275194 31587 raft_consensus.cc:3032] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:40.275717 31587 raft_consensus.cc:729] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:40.276494 31587 consensus_queue.cc:260] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.277840 31587 raft_consensus.cc:3037] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.282409 31690 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 3.
I20260806 03:41:40.283875 31414 leader_election.cc:302] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:40.284981 31587 raft_consensus.cc:2454] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 3.
I20260806 03:41:40.286084 31991 raft_consensus.cc:2781] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:40.286978 31991 raft_consensus.cc:686] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 3 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:40.287844 31991 consensus_queue.cc:237] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.294987 31319 catalog_manager.cc:5495] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 2 to 3, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:40.313671 31990 raft_consensus.cc:979] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: : Instructing follower a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:40.314129 31959 raft_consensus.cc:1067] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Signalling peer a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:40.315814 31690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:43916
I20260806 03:41:40.316322 31690 raft_consensus.cc:482] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:40.316812 31690 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.317845 31959 raft_consensus.cc:979] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.318342 31990 raft_consensus.cc:1067] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Signalling peer f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.319826 31463 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:46768
I20260806 03:41:40.320336 31463 raft_consensus.cc:482] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:40.320642 31463 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.323204 31690 raft_consensus.cc:504] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.326210 31959 raft_consensus.cc:979] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.326694 31959 raft_consensus.cc:1067] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Signalling peer f2aaabfd88234360b4fb0550085cac61 to start an election
I20260806 03:41:40.326973 31464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:40.326973 31463 raft_consensus.cc:504] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.327694 31464 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.327975 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1881a43bbca74cfba2311ba0e7fe2d0f" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:40.328531 31587 raft_consensus.cc:3032] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:40.328822 31587 raft_consensus.cc:729] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:40.329646 31468 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20"
dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
from {username='slave'} at 127.0.0.1:46768
I20260806 03:41:40.329572 31587 consensus_queue.cc:260] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.330196 31468 raft_consensus.cc:482] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:40.330752 31468 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:40.331065 31688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:40.331697 31688 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.332224 31583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb63622f25704c2e8c76cc0a5b577812" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:40.332830 31583 raft_consensus.cc:3032] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:41:40.333117 31583 raft_consensus.cc:729] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:40.333839 31583 consensus_queue.cc:260] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.335119 31583 raft_consensus.cc:3037] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.335446 31464 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 3.
I20260806 03:41:40.336861 31644 leader_election.cc:302] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:40.337905 31958 raft_consensus.cc:2781] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:40.330781 31587 raft_consensus.cc:3037] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:41:40.338581 31688 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 3.
I20260806 03:41:40.340029 31414 leader_election.cc:302] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:40.341183 31583 raft_consensus.cc:2454] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 3.
I20260806 03:41:40.341215 31991 raft_consensus.cc:2781] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:41:40.341950 31991 raft_consensus.cc:686] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 3 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:40.342072 31690 leader_election.cc:288] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 3 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:40.343142 31991 consensus_queue.cc:237] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.344211 31587 raft_consensus.cc:2454] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 3.
I20260806 03:41:40.345203 31468 raft_consensus.cc:504] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.347082 31468 leader_election.cc:288] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 4 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:40.348351 31463 leader_election.cc:288] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 3 election: Requested vote from peers 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407), a6e18afe5269433c825d5b598e2302e0 (127.30.138.67:38529)
I20260806 03:41:40.348546 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:40.349073 31587 raft_consensus.cc:3032] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:41:40.349359 31587 raft_consensus.cc:729] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:40.349466 31690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23aa35b40d44524b86a004bb1122b20" candidate_uuid: "f2aaabfd88234360b4fb0550085cac61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
I20260806 03:41:40.350152 31690 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:40.350212 31587 consensus_queue.cc:260] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.350943 31958 raft_consensus.cc:686] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:40.351643 31587 raft_consensus.cc:3037] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:41:40.351990 31958 consensus_queue.cc:237] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.357295 31587 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 4.
I20260806 03:41:40.358798 31415 leader_election.cc:302] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a69965c26904db58123d8b355f03ae2, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:40.359655 31992 raft_consensus.cc:2781] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:41:40.360296 31690 raft_consensus.cc:2454] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2aaabfd88234360b4fb0550085cac61 in term 4.
I20260806 03:41:40.361054 31992 raft_consensus.cc:686] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 4 LEADER]: Becoming Leader. State: Replica: f2aaabfd88234360b4fb0550085cac61, State: Running, Role: LEADER
I20260806 03:41:40.361640 31319 catalog_manager.cc:5495] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 2 to 3, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 3 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:40.362097 31992 consensus_queue.cc:237] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.374153 31319 catalog_manager.cc:5495] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 2 to 3, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:40.384985 31319 catalog_manager.cc:5495] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 reported cstate change: term changed from 3 to 4, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to f2aaabfd88234360b4fb0550085cac61 (127.30.138.65). New cstate: current_term: 4 leader_uuid: "f2aaabfd88234360b4fb0550085cac61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:40.655237 31990 raft_consensus.cc:979] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: : Instructing follower a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:40.655748 31959 raft_consensus.cc:1067] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Signalling peer a6e18afe5269433c825d5b598e2302e0 to start an election
I20260806 03:41:40.656936 31690 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3"
dest_uuid: "a6e18afe5269433c825d5b598e2302e0"
from {username='slave'} at 127.0.0.1:43916
I20260806 03:41:40.657387 31690 raft_consensus.cc:482] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:40.657672 31690 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:40.662019 31690 raft_consensus.cc:504] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.663542 31690 leader_election.cc:288] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Requested vote from peers f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451), 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407)
I20260806 03:41:40.664423 31463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f2aaabfd88234360b4fb0550085cac61"
I20260806 03:41:40.664580 31587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38c76ec03e104b158aac90e1fb81bbb3" candidate_uuid: "a6e18afe5269433c825d5b598e2302e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a69965c26904db58123d8b355f03ae2"
I20260806 03:41:40.665027 31463 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:40.665117 31587 raft_consensus.cc:3032] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:40.665481 31587 raft_consensus.cc:729] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a69965c26904db58123d8b355f03ae2, State: Running, Role: LEADER
I20260806 03:41:40.666157 31587 consensus_queue.cc:260] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.667292 31587 raft_consensus.cc:3037] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:40.669719 31463 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:40.670727 31644 leader_election.cc:302] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e18afe5269433c825d5b598e2302e0, f2aaabfd88234360b4fb0550085cac61; no voters:
I20260806 03:41:40.671545 31958 raft_consensus.cc:2781] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:40.671963 31587 raft_consensus.cc:2454] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e18afe5269433c825d5b598e2302e0 in term 2.
I20260806 03:41:40.672061 31958 raft_consensus.cc:686] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Becoming Leader. State: Replica: a6e18afe5269433c825d5b598e2302e0, State: Running, Role: LEADER
I20260806 03:41:40.672992 31958 consensus_queue.cc:237] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } }
I20260806 03:41:40.679327 31319 catalog_manager.cc:5495] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 reported cstate change: term changed from 1 to 2, leader changed from 0a69965c26904db58123d8b355f03ae2 (127.30.138.66) to a6e18afe5269433c825d5b598e2302e0 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "a6e18afe5269433c825d5b598e2302e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:40.701881 31587 raft_consensus.cc:1261] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.703033 31992 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:40.713165 31690 raft_consensus.cc:1261] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.714773 31991 consensus_queue.cc:1035] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:40.798628 31690 raft_consensus.cc:1261] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.799847 31991 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:40.809698 31587 raft_consensus.cc:1261] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.811899 32002 consensus_queue.cc:1035] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:41:40.821527 31463 raft_consensus.cc:1261] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.822719 31958 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:41:40.832182 31587 raft_consensus.cc:1261] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:41:40.834496 31958 consensus_queue.cc:1035] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:41:40.838160 31690 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:41:40.839344 31991 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e18afe5269433c825d5b598e2302e0" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 38529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:41:40.853251 31587 raft_consensus.cc:1261] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Refusing update from remote peer f2aaabfd88234360b4fb0550085cac61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:41:40.855064 31992 consensus_queue.cc:1035] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:41:41.089885 31463 raft_consensus.cc:1261] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:41.091472 32024 consensus_queue.cc:1035] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2aaabfd88234360b4fb0550085cac61" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:41.102283 31587 raft_consensus.cc:1261] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Refusing update from remote peer a6e18afe5269433c825d5b598e2302e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:41.103909 31958 consensus_queue.cc:1035] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a69965c26904db58123d8b355f03ae2" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:41.263620 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:41.268798 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:41.269261 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:42.270200 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:42.275194 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:42.275691 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:43.276661 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:43.281394 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:43.281884 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:44.282831 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:44.287855 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:44.288267 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:45.289165 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:45.294866 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:45.295341 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:46.296254 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:46.300866 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:46.301520 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:47.302850 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:47.307870 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:47.308290 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:48.309532 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:48.316736 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:48.317340 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:49.318436 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:49.323911 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:49.324565 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:50.354576 31273 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 03:41:50.355460 31273 tablet_server.cc:178] TabletServer@127.30.138.65:0 shutting down...
W20260806 03:41:50.358433 31529 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260806 03:41:50.362951 31529 consensus_peers.cc:487] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 -> Peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Couldn't send request to peer f2aaabfd88234360b4fb0550085cac61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:41:50.363523 31529 consensus_peers.cc:487] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 -> Peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Couldn't send request to peer f2aaabfd88234360b4fb0550085cac61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:41:50.391991 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:41:50.392822 31273 tablet_replica.cc:330] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.393654 31273 raft_consensus.cc:2229] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.394315 31273 raft_consensus.cc:2258] T 04868dda0d8343c9976e9c058425b828 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.396993 31273 tablet_replica.cc:330] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.397517 31273 raft_consensus.cc:2229] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.398458 31273 raft_consensus.cc:2258] T b23aa35b40d44524b86a004bb1122b20 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.400134 31273 tablet_replica.cc:330] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.400626 31273 raft_consensus.cc:2229] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.401463 31273 raft_consensus.cc:2258] T bb63622f25704c2e8c76cc0a5b577812 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.403188 31273 tablet_replica.cc:330] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.403714 31273 raft_consensus.cc:2229] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.404508 31273 raft_consensus.cc:2258] T 68a9cab51a0d48fbaf69e141590d7556 P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.406175 31273 tablet_replica.cc:330] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.406708 31273 raft_consensus.cc:2229] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.407140 31273 raft_consensus.cc:2258] T 38c76ec03e104b158aac90e1fb81bbb3 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.408684 31273 tablet_replica.cc:330] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.409148 31273 raft_consensus.cc:2229] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.409641 31273 raft_consensus.cc:2258] T 36108fe9044b420da02f5d5065c34021 P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.411376 31273 tablet_replica.cc:330] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.411837 31273 raft_consensus.cc:2229] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.412293 31273 raft_consensus.cc:2258] T f8925cebe2dc42fc8cacb7f8094fc8ae P f2aaabfd88234360b4fb0550085cac61 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.413865 31273 tablet_replica.cc:330] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.414307 31273 raft_consensus.cc:2229] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.414805 31273 raft_consensus.cc:2258] T 1881a43bbca74cfba2311ba0e7fe2d0f P f2aaabfd88234360b4fb0550085cac61 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.416757 31273 tablet_replica.cc:330] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61: stopping tablet replica
I20260806 03:41:50.417219 31273 raft_consensus.cc:2229] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.417814 31273 raft_consensus.cc:2258] T ff71bdd5817f4e58ab2f9f040e93bfd9 P f2aaabfd88234360b4fb0550085cac61 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260806 03:41:50.545459 31529 consensus_peers.cc:487] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 -> Peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Couldn't send request to peer f2aaabfd88234360b4fb0550085cac61. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:45451: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:41:50.545982 31273 tablet_server.cc:194] TabletServer@127.30.138.65:0 shutdown complete.
W20260806 03:41:50.556080 31644 consensus_peers.cc:487] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 -> Peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Couldn't send request to peer f2aaabfd88234360b4fb0550085cac61. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:45451: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:41:50.567184 31273 tablet_server.cc:178] TabletServer@127.30.138.66:0 shutting down...
I20260806 03:41:50.594125 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:41:50.594964 31273 tablet_replica.cc:330] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.595817 31273 raft_consensus.cc:2229] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.596463 31273 raft_consensus.cc:2258] T 38c76ec03e104b158aac90e1fb81bbb3 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.599221 31273 tablet_replica.cc:330] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.599957 31273 raft_consensus.cc:2229] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.600488 31273 raft_consensus.cc:2258] T 1881a43bbca74cfba2311ba0e7fe2d0f P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.602689 31273 tablet_replica.cc:330] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.603374 31273 raft_consensus.cc:2229] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.604362 31273 raft_consensus.cc:2258] T f8925cebe2dc42fc8cacb7f8094fc8ae P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.606415 31273 tablet_replica.cc:330] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.607030 31273 raft_consensus.cc:2229] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.607772 31273 raft_consensus.cc:2258] T ff71bdd5817f4e58ab2f9f040e93bfd9 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.609197 31273 tablet_replica.cc:330] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.609642 31273 raft_consensus.cc:2229] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.609993 31273 raft_consensus.cc:2258] T bb63622f25704c2e8c76cc0a5b577812 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.611567 31273 tablet_replica.cc:330] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.611968 31273 raft_consensus.cc:2229] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.612311 31273 raft_consensus.cc:2258] T b23aa35b40d44524b86a004bb1122b20 P 0a69965c26904db58123d8b355f03ae2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.613781 31273 tablet_replica.cc:330] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.614178 31273 raft_consensus.cc:2229] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.614960 31273 raft_consensus.cc:2258] T 04868dda0d8343c9976e9c058425b828 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.616396 31273 tablet_replica.cc:330] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.616863 31273 raft_consensus.cc:2229] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.617353 31273 raft_consensus.cc:2258] T 68a9cab51a0d48fbaf69e141590d7556 P 0a69965c26904db58123d8b355f03ae2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.618983 31273 tablet_replica.cc:330] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2: stopping tablet replica
I20260806 03:41:50.619400 31273 raft_consensus.cc:2229] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.619835 31273 raft_consensus.cc:2258] T 36108fe9044b420da02f5d5065c34021 P 0a69965c26904db58123d8b355f03ae2 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 03:41:50.632675 31645 consensus_peers.cc:487] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 -> Peer 0a69965c26904db58123d8b355f03ae2 (127.30.138.66:39407): Couldn't send request to peer 0a69965c26904db58123d8b355f03ae2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:41:50.649276 31273 tablet_server.cc:194] TabletServer@127.30.138.66:0 shutdown complete.
I20260806 03:41:50.668161 31273 tablet_server.cc:178] TabletServer@127.30.138.67:0 shutting down...
W20260806 03:41:50.695763 31644 consensus_peers.cc:487] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 -> Peer f2aaabfd88234360b4fb0550085cac61 (127.30.138.65:45451): Couldn't send request to peer f2aaabfd88234360b4fb0550085cac61. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:45451: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:41:50.698616 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:41:50.699349 31273 tablet_replica.cc:330] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.700325 31273 raft_consensus.cc:2229] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.701305 31273 raft_consensus.cc:2258] T 38c76ec03e104b158aac90e1fb81bbb3 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.703001 31273 tablet_replica.cc:330] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.703606 31273 raft_consensus.cc:2229] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.704314 31273 raft_consensus.cc:2258] T 36108fe9044b420da02f5d5065c34021 P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.705879 31273 tablet_replica.cc:330] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.706343 31273 raft_consensus.cc:2229] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.706830 31273 raft_consensus.cc:2258] T bb63622f25704c2e8c76cc0a5b577812 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.708253 31273 tablet_replica.cc:330] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.708860 31273 raft_consensus.cc:2229] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.709270 31273 raft_consensus.cc:2258] T b23aa35b40d44524b86a004bb1122b20 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.710913 31273 tablet_replica.cc:330] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.711354 31273 raft_consensus.cc:2229] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.711709 31273 raft_consensus.cc:2258] T 04868dda0d8343c9976e9c058425b828 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.713230 31273 tablet_replica.cc:330] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.713676 31273 raft_consensus.cc:2229] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.714205 31273 raft_consensus.cc:2258] T f8925cebe2dc42fc8cacb7f8094fc8ae P a6e18afe5269433c825d5b598e2302e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.715968 31273 tablet_replica.cc:330] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.716391 31273 raft_consensus.cc:2229] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:41:50.717192 31273 raft_consensus.cc:2258] T 1881a43bbca74cfba2311ba0e7fe2d0f P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.718851 31273 tablet_replica.cc:330] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.719262 31273 raft_consensus.cc:2229] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.719585 31273 raft_consensus.cc:2258] T ff71bdd5817f4e58ab2f9f040e93bfd9 P a6e18afe5269433c825d5b598e2302e0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.721089 31273 tablet_replica.cc:330] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0: stopping tablet replica
I20260806 03:41:50.721457 31273 raft_consensus.cc:2229] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:41:50.721808 31273 raft_consensus.cc:2258] T 68a9cab51a0d48fbaf69e141590d7556 P a6e18afe5269433c825d5b598e2302e0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:50.749362 31273 tablet_server.cc:194] TabletServer@127.30.138.67:0 shutdown complete.
I20260806 03:41:50.766992 31273 master.cc:537] Master@127.30.138.126:41693 shutting down...
I20260806 03:41:51.113672 31273 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:41:51.114276 31273 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:41:51.114701 31273 tablet_replica.cc:330] T 00000000000000000000000000000000 P 788391b0417d4284b4ccf9781bfdb3b5: stopping tablet replica
I20260806 03:41:51.134228 31273 master.cc:552] Master@127.30.138.126:41693 shutdown complete.
0a69965c26904db58123d8b355f03ae2 4
a6e18afe5269433c825d5b598e2302e0 1
f2aaabfd88234360b4fb0550085cac61 4
0a69965c26904db58123d8b355f03ae2 3
a6e18afe5269433c825d5b598e2302e0 3
f2aaabfd88234360b4fb0550085cac61 3
0a69965c26904db58123d8b355f03ae2 8
a6e18afe5269433c825d5b598e2302e0 1
f2aaabfd88234360b4fb0550085cac61 0
0a69965c26904db58123d8b355f03ae2 3
a6e18afe5269433c825d5b598e2302e0 3
f2aaabfd88234360b4fb0550085cac61 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31723 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 03:41:51.176740 31273 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.138.126:44755
I20260806 03:41:51.177878 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:51.183671 32064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.184028 32065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:51.185528 31273 server_base.cc:1034] running on GCE node
W20260806 03:41:51.185984 32068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:51.186986 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:51.187201 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:51.187311 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987711187303 us; error 0 us; skew 500 ppm
I20260806 03:41:51.187812 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:51.193617 31273 webserver.cc:460] Webserver started at http://127.30.138.126:42411/ using document root <none> and password file <none>
I20260806 03:41:51.194109 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:51.194262 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:51.194584 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:51.195779 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/master-0-root/instance:
uuid: "03227a8f28c24f50aa93577c96b6e773"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.200600 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 03:41:51.203958 32075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.204770 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 03:41:51.205065 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/master-0-root
uuid: "03227a8f28c24f50aa93577c96b6e773"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.205375 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:51.217618 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:51.219398 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:51.266472 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.126:44755
I20260806 03:41:51.266644 32157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.126:44755 every 8 connection(s)
I20260806 03:41:51.270501 32159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.280834 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773: Bootstrap starting.
I20260806 03:41:51.285784 32159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:51.290510 32159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773: No bootstrap required, opened a new log
I20260806 03:41:51.292689 32159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER }
I20260806 03:41:51.293191 32159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:51.293380 32159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03227a8f28c24f50aa93577c96b6e773, State: Initialized, Role: FOLLOWER
I20260806 03:41:51.293959 32159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER }
I20260806 03:41:51.294705 32159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:41:51.294950 32159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:41:51.295218 32159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:51.299738 32159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER }
I20260806 03:41:51.300580 32159 leader_election.cc:302] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03227a8f28c24f50aa93577c96b6e773; no voters:
I20260806 03:41:51.301847 32159 leader_election.cc:288] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:41:51.302291 32163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:51.303789 32163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 1 LEADER]: Becoming Leader. State: Replica: 03227a8f28c24f50aa93577c96b6e773, State: Running, Role: LEADER
I20260806 03:41:51.304733 32163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER }
I20260806 03:41:51.305461 32159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:41:51.311406 32164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03227a8f28c24f50aa93577c96b6e773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER } }
I20260806 03:41:51.311379 32165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03227a8f28c24f50aa93577c96b6e773. Latest consensus state: current_term: 1 leader_uuid: "03227a8f28c24f50aa93577c96b6e773" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03227a8f28c24f50aa93577c96b6e773" member_type: VOTER } }
I20260806 03:41:51.312132 32165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [sys.catalog]: This master's current role is: LEADER
I20260806 03:41:51.312130 32164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [sys.catalog]: This master's current role is: LEADER
I20260806 03:41:51.314925 32175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:41:51.320242 32175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:41:51.323114 31273 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:41:51.329627 32175 catalog_manager.cc:1347] Generated new cluster ID: 308f5e7af3c443e39dfb3e5979d68d78
I20260806 03:41:51.330025 32175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:41:51.351405 32175 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:41:51.352991 32175 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:41:51.368858 32175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773: Generated new TSK 0
I20260806 03:41:51.369545 32175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:41:51.394016 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:51.401259 32191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.401372 32193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.403507 32195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:51.405907 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:51.407186 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:51.407413 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:51.407595 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987711407585 us; error 0 us; skew 500 ppm
I20260806 03:41:51.408156 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:51.410686 31273 webserver.cc:460] Webserver started at http://127.30.138.65:42419/ using document root <none> and password file <none>
I20260806 03:41:51.411219 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:51.411417 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:51.411698 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:51.412810 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-0-root/instance:
uuid: "0bc3e76ecad34049bbecde2881393966"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.417665 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 03:41:51.421116 32203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.421983 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 03:41:51.422325 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
uuid: "0bc3e76ecad34049bbecde2881393966"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.422797 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:51.441105 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:51.442489 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:51.444116 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:51.447162 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:51.447388 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.447645 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:51.447831 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.489426 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.65:38911
I20260806 03:41:51.489549 32292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.65:38911 every 8 connection(s)
I20260806 03:41:51.494063 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:51.501812 32299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.504662 32300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.505851 32302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:51.506479 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:51.507807 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:51.508023 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:51.508211 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987711508196 us; error 0 us; skew 500 ppm
I20260806 03:41:51.508904 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:51.510411 32293 heartbeater.cc:347] Connected to a master server at 127.30.138.126:44755
I20260806 03:41:51.510842 32293 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:51.511550 32293 heartbeater.cc:511] Master 127.30.138.126:44755 requested a full tablet report, sending...
I20260806 03:41:51.511932 31273 webserver.cc:460] Webserver started at http://127.30.138.66:41497/ using document root <none> and password file <none>
I20260806 03:41:51.512547 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:51.512835 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:51.513119 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:51.513736 32102 ts_manager.cc:194] Registered new tserver with Master: 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911)
I20260806 03:41:51.514423 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-1-root/instance:
uuid: "d558039cc1914d328d0e3f9e637d66f1"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.515735 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45954
I20260806 03:41:51.520104 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 03:41:51.523501 32313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.524410 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 03:41:51.524665 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
uuid: "d558039cc1914d328d0e3f9e637d66f1"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.524911 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:51.542793 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:51.543943 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:51.545379 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:51.547629 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:51.547850 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.548084 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:51.548249 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.588945 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.66:39973
I20260806 03:41:51.589071 32405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.66:39973 every 8 connection(s)
I20260806 03:41:51.593850 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:41:51.600793 32413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.602483 32414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:41:51.604678 32418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:41:51.604947 31273 server_base.cc:1034] running on GCE node
I20260806 03:41:51.606123 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:41:51.606491 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:41:51.606707 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987711606690 us; error 0 us; skew 500 ppm
I20260806 03:41:51.607465 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:41:51.608191 32406 heartbeater.cc:347] Connected to a master server at 127.30.138.126:44755
I20260806 03:41:51.608628 32406 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:51.609411 32406 heartbeater.cc:511] Master 127.30.138.126:44755 requested a full tablet report, sending...
I20260806 03:41:51.610191 31273 webserver.cc:460] Webserver started at http://127.30.138.67:33093/ using document root <none> and password file <none>
I20260806 03:41:51.610837 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:41:51.611070 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:41:51.611349 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:41:51.611630 32102 ts_manager.cc:194] Registered new tserver with Master: d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:51.612588 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-2-root/instance:
uuid: "89cc456269ab41c98e1b3f7813c490d8"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.613183 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45958
I20260806 03:41:51.618137 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:41:51.621619 32423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.622538 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 03:41:51.622840 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
uuid: "89cc456269ab41c98e1b3f7813c490d8"
format_stamp: "Formatted at 2026-08-06 03:41:51 on dist-test-slave-tk9z"
I20260806 03:41:51.623202 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785987679226243-31273-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:41:51.640221 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:41:51.641554 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:41:51.643136 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:41:51.645359 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:41:51.645596 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.645838 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:41:51.646005 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:41:51.687542 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.67:40459
I20260806 03:41:51.687690 32516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.67:40459 every 8 connection(s)
I20260806 03:41:51.703125 32517 heartbeater.cc:347] Connected to a master server at 127.30.138.126:44755
I20260806 03:41:51.703573 32517 heartbeater.cc:464] Registering TS with master...
I20260806 03:41:51.704310 32517 heartbeater.cc:511] Master 127.30.138.126:44755 requested a full tablet report, sending...
I20260806 03:41:51.706188 32102 ts_manager.cc:194] Registered new tserver with Master: 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:51.706573 31273 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015984571s
I20260806 03:41:51.707995 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45974
I20260806 03:41:51.709008 31273 test_util.cc:268] Using random seed: -1423888588
I20260806 03:41:51.734936 32102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45990:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:41:51.737879 32102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:41:51.812714 32241 tablet_service.cc:1467] Processing CreateTablet for tablet 0eaba71e2dc24ac39581c7271f5c201e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:41:51.812986 32240 tablet_service.cc:1467] Processing CreateTablet for tablet 7e822611997a47e8bc60af35416dfddf (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:41:51.814769 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eaba71e2dc24ac39581c7271f5c201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.815006 32239 tablet_service.cc:1467] Processing CreateTablet for tablet e47b707c5ab64b28abac16902d9fff86 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:41:51.816927 32237 tablet_service.cc:1467] Processing CreateTablet for tablet eb1cfe35f4f645e4af266acefe4269e0 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:41:51.818615 32235 tablet_service.cc:1467] Processing CreateTablet for tablet 3f813cc1db34443f9e7525785bfebd1e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:41:51.819186 32239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47b707c5ab64b28abac16902d9fff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.819952 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f813cc1db34443f9e7525785bfebd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.823686 32234 tablet_service.cc:1467] Processing CreateTablet for tablet eb542237f30b4f9282429bc416260fa7 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:41:51.823928 32230 tablet_service.cc:1467] Processing CreateTablet for tablet d446cdb829c1455799c42cf1d006f617 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:41:51.825145 32234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb542237f30b4f9282429bc416260fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.826224 32229 tablet_service.cc:1467] Processing CreateTablet for tablet cd12335f0f8e4005a88ac057dda964fc (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:41:51.827567 32231 tablet_service.cc:1467] Processing CreateTablet for tablet e10bdb723a5f47daa1fb1990ffb2b4b1 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:41:51.815006 32233 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0c226c59e6412cbfc08cf7b7e7c577 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:41:51.829095 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd12335f0f8e4005a88ac057dda964fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.834769 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0c226c59e6412cbfc08cf7b7e7c577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.835552 32240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e822611997a47e8bc60af35416dfddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.836408 32231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10bdb723a5f47daa1fb1990ffb2b4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.838928 32237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cfe35f4f645e4af266acefe4269e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.840206 32230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d446cdb829c1455799c42cf1d006f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.876816 32351 tablet_service.cc:1467] Processing CreateTablet for tablet 0eaba71e2dc24ac39581c7271f5c201e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:41:51.878216 32351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eaba71e2dc24ac39581c7271f5c201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.879424 32348 tablet_service.cc:1467] Processing CreateTablet for tablet eb1cfe35f4f645e4af266acefe4269e0 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:41:51.880779 32348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cfe35f4f645e4af266acefe4269e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.883587 32346 tablet_service.cc:1467] Processing CreateTablet for tablet eb542237f30b4f9282429bc416260fa7 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:41:51.884915 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb542237f30b4f9282429bc416260fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.885737 32349 tablet_service.cc:1467] Processing CreateTablet for tablet e47b707c5ab64b28abac16902d9fff86 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:41:51.887087 32349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47b707c5ab64b28abac16902d9fff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.887591 32344 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0c226c59e6412cbfc08cf7b7e7c577 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:41:51.888878 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0c226c59e6412cbfc08cf7b7e7c577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.883591 32350 tablet_service.cc:1467] Processing CreateTablet for tablet 7e822611997a47e8bc60af35416dfddf (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:41:51.891113 32347 tablet_service.cc:1467] Processing CreateTablet for tablet 3f813cc1db34443f9e7525785bfebd1e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:41:51.891784 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e822611997a47e8bc60af35416dfddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.892416 32347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f813cc1db34443f9e7525785bfebd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.894479 32534 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:51.900926 32534 tablet_bootstrap.cc:654] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:51.904192 32343 tablet_service.cc:1467] Processing CreateTablet for tablet e10bdb723a5f47daa1fb1990ffb2b4b1 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:41:51.905596 32343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10bdb723a5f47daa1fb1990ffb2b4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.908138 32340 tablet_service.cc:1467] Processing CreateTablet for tablet d446cdb829c1455799c42cf1d006f617 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:41:51.909466 32340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d446cdb829c1455799c42cf1d006f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.912099 32339 tablet_service.cc:1467] Processing CreateTablet for tablet cd12335f0f8e4005a88ac057dda964fc (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:41:51.913407 32339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd12335f0f8e4005a88ac057dda964fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.931895 32536 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:51.938153 32465 tablet_service.cc:1467] Processing CreateTablet for tablet 0eaba71e2dc24ac39581c7271f5c201e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:41:51.939637 32465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0eaba71e2dc24ac39581c7271f5c201e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.943141 32460 tablet_service.cc:1467] Processing CreateTablet for tablet 3f813cc1db34443f9e7525785bfebd1e (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:41:51.944522 32460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f813cc1db34443f9e7525785bfebd1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.947599 32457 tablet_service.cc:1467] Processing CreateTablet for tablet eb542237f30b4f9282429bc416260fa7 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:41:51.948963 32457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb542237f30b4f9282429bc416260fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.951795 32455 tablet_service.cc:1467] Processing CreateTablet for tablet 3f0c226c59e6412cbfc08cf7b7e7c577 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:41:51.953086 32455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f0c226c59e6412cbfc08cf7b7e7c577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.955628 32454 tablet_service.cc:1467] Processing CreateTablet for tablet e10bdb723a5f47daa1fb1990ffb2b4b1 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:41:51.956969 32454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e10bdb723a5f47daa1fb1990ffb2b4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.959574 32463 tablet_service.cc:1467] Processing CreateTablet for tablet e47b707c5ab64b28abac16902d9fff86 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:41:51.960930 32463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47b707c5ab64b28abac16902d9fff86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.964437 32462 tablet_service.cc:1467] Processing CreateTablet for tablet eb1cfe35f4f645e4af266acefe4269e0 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:41:51.965859 32462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1cfe35f4f645e4af266acefe4269e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.968385 32464 tablet_service.cc:1467] Processing CreateTablet for tablet 7e822611997a47e8bc60af35416dfddf (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:41:51.970252 32464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e822611997a47e8bc60af35416dfddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.971827 32453 tablet_service.cc:1467] Processing CreateTablet for tablet d446cdb829c1455799c42cf1d006f617 (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:41:51.973197 32453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d446cdb829c1455799c42cf1d006f617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.969933 32452 tablet_service.cc:1467] Processing CreateTablet for tablet cd12335f0f8e4005a88ac057dda964fc (DEFAULT_TABLE table=test-workload [id=0c650ac314144402ba49329daeb3c8ae]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:41:51.977064 32452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd12335f0f8e4005a88ac057dda964fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:41:51.978927 32534 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:51.979508 32534 ts_tablet_manager.cc:1397] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.085s user 0.024s sys 0.049s
I20260806 03:41:51.982399 32534 raft_consensus.cc:348] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:51.983326 32534 raft_consensus.cc:374] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:51.983690 32534 raft_consensus.cc:729] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:51.984910 32536 tablet_bootstrap.cc:654] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:51.984472 32534 consensus_queue.cc:260] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.068027 32534 ts_tablet_manager.cc:1428] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.088s user 0.017s sys 0.039s
W20260806 03:41:52.068522 32080 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.30.138.66:39973, user_credentials={real_user=slave}} blocked reactor thread for 68136.7us
I20260806 03:41:52.068939 32293 heartbeater.cc:503] Master 127.30.138.126:44755 was elected leader, sending a full tablet report...
I20260806 03:41:52.069139 32534 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.077718 32534 tablet_bootstrap.cc:654] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.114671 32538 raft_consensus.cc:482] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.115447 32538 raft_consensus.cc:504] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.130852 32536 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.132388 32536 ts_tablet_manager.cc:1397] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.201s user 0.020s sys 0.018s
I20260806 03:41:52.136013 32536 raft_consensus.cc:348] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.137737 32536 raft_consensus.cc:374] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.138077 32536 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.139261 32534 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.139750 32534 ts_tablet_manager.cc:1397] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.071s user 0.018s sys 0.013s
I20260806 03:41:52.138973 32536 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.143281 32534 raft_consensus.cc:348] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.144374 32534 raft_consensus.cc:374] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.144739 32534 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.145630 32534 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.148356 32534 ts_tablet_manager.cc:1428] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:41:52.143307 32542 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.149502 32534 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.149789 32538 leader_election.cc:288] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.156978 32534 tablet_bootstrap.cc:654] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.164472 32542 tablet_bootstrap.cc:654] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.173233 32406 heartbeater.cc:503] Master 127.30.138.126:44755 was elected leader, sending a full tablet report...
I20260806 03:41:52.173834 32536 ts_tablet_manager.cc:1428] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.041s user 0.017s sys 0.003s
I20260806 03:41:52.174934 32536 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.181274 32536 tablet_bootstrap.cc:654] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.191272 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
W20260806 03:41:52.193742 32206 leader_election.cc:341] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.199188 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
W20260806 03:41:52.201581 32207 leader_election.cc:341] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.202282 32207 leader_election.cc:302] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966; no voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.206907 32538 raft_consensus.cc:2726] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.209973 32534 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.210018 32542 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.210675 32534 ts_tablet_manager.cc:1397] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.061s user 0.030s sys 0.027s
I20260806 03:41:52.210722 32542 ts_tablet_manager.cc:1397] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.003s
I20260806 03:41:52.211668 32536 tablet_bootstrap.cc:492] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.212162 32536 ts_tablet_manager.cc:1397] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.016s
I20260806 03:41:52.213255 32534 raft_consensus.cc:348] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.213848 32534 raft_consensus.cc:374] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.214247 32534 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.213788 32542 raft_consensus.cc:348] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.214715 32542 raft_consensus.cc:374] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.215049 32542 raft_consensus.cc:729] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.214813 32536 raft_consensus.cc:348] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.215109 32534 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.215555 32536 raft_consensus.cc:374] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.215914 32536 raft_consensus.cc:729] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.217166 32534 ts_tablet_manager.cc:1428] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 03:41:52.216590 32536 consensus_queue.cc:260] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.218012 32534 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.218855 32542 consensus_queue.cc:260] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.223246 32536 ts_tablet_manager.cc:1428] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 03:41:52.224291 32536 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.226265 32534 tablet_bootstrap.cc:654] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.235539 32517 heartbeater.cc:503] Master 127.30.138.126:44755 was elected leader, sending a full tablet report...
I20260806 03:41:52.236212 32542 ts_tablet_manager.cc:1428] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.025s user 0.015s sys 0.005s
I20260806 03:41:52.237258 32542 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.231416 32536 tablet_bootstrap.cc:654] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.243211 32542 tablet_bootstrap.cc:654] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.243475 32536 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.245303 32536 ts_tablet_manager.cc:1397] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.002s
I20260806 03:41:52.247620 32534 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.248225 32534 ts_tablet_manager.cc:1397] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.006s
I20260806 03:41:52.248711 32536 raft_consensus.cc:348] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.249728 32536 raft_consensus.cc:374] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.250105 32536 raft_consensus.cc:729] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.251196 32534 raft_consensus.cc:348] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.251192 32536 consensus_queue.cc:260] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.252113 32534 raft_consensus.cc:374] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.252549 32534 raft_consensus.cc:729] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.253306 32534 consensus_queue.cc:260] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.258287 32534 ts_tablet_manager.cc:1428] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 03:41:52.259387 32534 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.261368 32536 ts_tablet_manager.cc:1428] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.016s user 0.008s sys 0.001s
I20260806 03:41:52.265657 32534 tablet_bootstrap.cc:654] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.270506 32536 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.270774 32542 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.271509 32542 ts_tablet_manager.cc:1397] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.012s
I20260806 03:41:52.275084 32542 raft_consensus.cc:348] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.275584 32536 tablet_bootstrap.cc:654] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.275707 32542 raft_consensus.cc:374] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.276088 32542 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.276814 32542 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.277375 32534 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.277926 32534 ts_tablet_manager.cc:1397] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.019s user 0.017s sys 0.000s
I20260806 03:41:52.279095 32542 ts_tablet_manager.cc:1428] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:41:52.280241 32542 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.280689 32534 raft_consensus.cc:348] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.281464 32534 raft_consensus.cc:374] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.281785 32534 raft_consensus.cc:729] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.282613 32534 consensus_queue.cc:260] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.285707 32534 ts_tablet_manager.cc:1428] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:41:52.287246 32534 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.288672 32542 tablet_bootstrap.cc:654] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.293960 32534 tablet_bootstrap.cc:654] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.303498 32536 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.303565 32542 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.304126 32542 ts_tablet_manager.cc:1397] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.002s
I20260806 03:41:52.304101 32536 ts_tablet_manager.cc:1397] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.012s
I20260806 03:41:52.304478 32534 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.304979 32534 ts_tablet_manager.cc:1397] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.004s
I20260806 03:41:52.307214 32542 raft_consensus.cc:348] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.307515 32536 raft_consensus.cc:348] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.307996 32542 raft_consensus.cc:374] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.308290 32536 raft_consensus.cc:374] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.308336 32542 raft_consensus.cc:729] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.308139 32534 raft_consensus.cc:348] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.308696 32536 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.308923 32534 raft_consensus.cc:374] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.309244 32534 raft_consensus.cc:729] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.309342 32542 consensus_queue.cc:260] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.309437 32536 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.309926 32534 consensus_queue.cc:260] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.312196 32536 ts_tablet_manager.cc:1428] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 03:41:52.312378 32534 ts_tablet_manager.cc:1428] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:41:52.313170 32536 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.314522 32542 ts_tablet_manager.cc:1428] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260806 03:41:52.315457 32542 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.313172 32534 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
W20260806 03:41:52.318331 32177 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:41:52.321595 32542 tablet_bootstrap.cc:654] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.323666 32534 tablet_bootstrap.cc:654] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.327018 32548 raft_consensus.cc:482] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.327513 32548 raft_consensus.cc:504] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.329970 32548 leader_election.cc:288] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.330294 32536 tablet_bootstrap.cc:654] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.346149 32552 raft_consensus.cc:482] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.346885 32552 raft_consensus.cc:504] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.361492 32552 leader_election.cc:288] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:52.364528 32534 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.365150 32534 ts_tablet_manager.cc:1397] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.005s
I20260806 03:41:52.367429 32552 raft_consensus.cc:482] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.367993 32552 raft_consensus.cc:504] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.370283 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.371230 32253 raft_consensus.cc:2454] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 0.
I20260806 03:41:52.373025 32314 leader_election.cc:302] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.368952 32534 raft_consensus.cc:348] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.374969 32534 raft_consensus.cc:374] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.375020 32548 raft_consensus.cc:2781] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.375614 32548 raft_consensus.cc:482] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.376086 32548 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.384109 32536 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.370081 32552 leader_election.cc:288] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:52.384667 32536 ts_tablet_manager.cc:1397] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.072s user 0.018s sys 0.010s
I20260806 03:41:52.385677 32542 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.386278 32542 ts_tablet_manager.cc:1397] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.040s
I20260806 03:41:52.385627 32548 raft_consensus.cc:504] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.388743 32548 leader_election.cc:288] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.375519 32534 raft_consensus.cc:729] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.389436 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:52.389192 32542 raft_consensus.cc:348] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.390240 32253 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.390142 32534 consensus_queue.cc:260] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.390218 32542 raft_consensus.cc:374] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.395228 32542 raft_consensus.cc:729] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.396119 32542 consensus_queue.cc:260] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.396669 32534 ts_tablet_manager.cc:1428] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.031s user 0.008s sys 0.003s
I20260806 03:41:52.397878 32534 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.398731 32542 ts_tablet_manager.cc:1428] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 03:41:52.399060 32253 raft_consensus.cc:2454] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 1.
I20260806 03:41:52.399749 32542 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.400270 32314 leader_election.cc:302] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.401145 32548 raft_consensus.cc:2781] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.404498 32534 tablet_bootstrap.cc:654] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.387492 32536 raft_consensus.cc:348] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.408171 32536 raft_consensus.cc:374] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.408535 32542 tablet_bootstrap.cc:654] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.408573 32536 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.413964 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.414536 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.415530 32252 raft_consensus.cc:2379] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89cc456269ab41c98e1b3f7813c490d8 in current term 1: Already voted for candidate d558039cc1914d328d0e3f9e637d66f1 in this term.
W20260806 03:41:52.415951 32425 leader_election.cc:341] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.420914 32552 raft_consensus.cc:482] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.421550 32552 raft_consensus.cc:504] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.426184 32552 leader_election.cc:288] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:52.409457 32536 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.429700 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.430802 32542 tablet_bootstrap.cc:492] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
W20260806 03:41:52.431034 32425 leader_election.cc:341] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.431447 32542 ts_tablet_manager.cc:1397] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20260806 03:41:52.434587 32548 raft_consensus.cc:686] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:52.436828 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.436625 32542 raft_consensus.cc:348] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.437361 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.437085 32548 consensus_queue.cc:237] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.437476 32479 raft_consensus.cc:2454] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 0.
I20260806 03:41:52.437391 32542 raft_consensus.cc:374] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.439646 32542 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.440357 32542 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.444062 32365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.444527 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.445436 32534 tablet_bootstrap.cc:492] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.445384 32366 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 0.
I20260806 03:41:52.445564 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.446074 32534 ts_tablet_manager.cc:1397] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.014s
I20260806 03:41:52.447640 32429 leader_election.cc:302] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8; no voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.448515 32536 ts_tablet_manager.cc:1428] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.064s user 0.012s sys 0.004s
I20260806 03:41:52.449316 32536 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.449046 32534 raft_consensus.cc:348] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.449978 32534 raft_consensus.cc:374] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.450332 32534 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.450824 32552 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.451045 32534 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.453637 32534 ts_tablet_manager.cc:1428] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:41:52.454660 32534 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.460199 32552 raft_consensus.cc:2726] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.461220 32534 tablet_bootstrap.cc:654] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.462296 32542 ts_tablet_manager.cc:1428] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.030s user 0.004s sys 0.005s
I20260806 03:41:52.463356 32542 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.466243 32538 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.466727 32538 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.468732 32538 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.471227 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.471920 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.472095 32363 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
W20260806 03:41:52.473289 32207 leader_election.cc:341] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.473613 32542 tablet_bootstrap.cc:654] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.473968 32206 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters: 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.475188 32538 raft_consensus.cc:2781] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.475606 32538 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.475951 32538 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.477360 32101 catalog_manager.cc:5495] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.489818 32537 raft_consensus.cc:482] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.490523 32537 raft_consensus.cc:504] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.493347 32537 leader_election.cc:288] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.495497 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.496345 32478 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.501804 32207 leader_election.cc:302] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:52.504339 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.505208 32363 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.509819 32537 raft_consensus.cc:2781] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.510293 32537 raft_consensus.cc:482] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.510968 32537 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.515925 32534 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.516498 32534 ts_tablet_manager.cc:1397] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.062s user 0.038s sys 0.021s
I20260806 03:41:52.527731 32534 raft_consensus.cc:348] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.528439 32534 raft_consensus.cc:374] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.528702 32534 raft_consensus.cc:729] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.529310 32534 consensus_queue.cc:260] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.533111 32538 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.537346 32536 tablet_bootstrap.cc:654] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.540673 32587 raft_consensus.cc:482] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.541312 32587 raft_consensus.cc:504] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
W20260806 03:41:52.544615 32429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:39973, user_credentials={real_user=slave}} blocked reactor thread for 97202.6us
I20260806 03:41:52.545606 32429 leader_election.cc:302] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters: 0bc3e76ecad34049bbecde2881393966
I20260806 03:41:52.546183 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.546959 32363 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:41:52.547099 32429 leader_election.cc:341] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.547749 32429 leader_election.cc:302] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8; no voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.548708 32552 raft_consensus.cc:2781] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.549099 32552 raft_consensus.cc:482] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.549418 32552 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.550881 32534 ts_tablet_manager.cc:1428] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.034s user 0.020s sys 0.013s
I20260806 03:41:52.552321 32534 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: Bootstrap starting.
I20260806 03:41:52.556057 32552 raft_consensus.cc:504] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.559010 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:52.559348 32534 tablet_bootstrap.cc:654] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.559628 32253 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.566762 32253 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 1.
I20260806 03:41:52.567905 32425 leader_election.cc:302] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:52.569332 32363 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.570848 32206 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.572130 32595 raft_consensus.cc:2781] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.575583 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.576406 32366 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.577584 32581 raft_consensus.cc:2726] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.581516 32537 raft_consensus.cc:504] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.582971 32581 raft_consensus.cc:2781] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.583561 32581 raft_consensus.cc:686] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:41:52.584636 32581 consensus_queue.cc:237] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.588711 32366 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 1.
I20260806 03:41:52.592254 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
W20260806 03:41:52.595326 32207 leader_election.cc:341] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.596356 32552 leader_election.cc:288] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:52.599553 32595 raft_consensus.cc:686] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.600816 32595 consensus_queue.cc:237] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.602900 32603 raft_consensus.cc:482] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.603411 32603 raft_consensus.cc:504] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.593459 32538 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.607729 32587 leader_election.cc:288] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.608371 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.609041 32478 raft_consensus.cc:2454] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.610286 32207 leader_election.cc:302] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:52.607232 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.611740 32536 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
W20260806 03:41:52.612141 32206 leader_election.cc:341] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.613253 32602 raft_consensus.cc:2781] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.613806 32602 raft_consensus.cc:482] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.615043 32602 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.614531 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.616410 32537 leader_election.cc:288] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.612212 32536 ts_tablet_manager.cc:1397] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.163s user 0.010s sys 0.001s
I20260806 03:41:52.617583 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.618160 32478 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:41:52.619320 32425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:38911, user_credentials={real_user=slave}} blocked reactor thread for 51501.2us
I20260806 03:41:52.616205 32366 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.625977 32478 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.627359 32207 leader_election.cc:302] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:52.632690 32102 catalog_manager.cc:5495] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.634879 32536 raft_consensus.cc:348] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.635591 32536 raft_consensus.cc:374] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.635874 32536 raft_consensus.cc:729] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.636423 32537 raft_consensus.cc:2781] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.632824 32602 raft_consensus.cc:504] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.641860 32542 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.642894 32603 leader_election.cc:288] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.643776 32365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.643756 32536 consensus_queue.cc:260] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.644034 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
W20260806 03:41:52.646605 32206 leader_election.cc:341] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.647557 32365 raft_consensus.cc:2454] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.644273 32537 raft_consensus.cc:686] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.649194 32537 consensus_queue.cc:237] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.649945 32366 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.650228 32542 ts_tablet_manager.cc:1397] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.187s user 0.057s sys 0.107s
I20260806 03:41:52.653115 32542 raft_consensus.cc:348] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.652902 32101 catalog_manager.cc:5495] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:52.654124 32534 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: No bootstrap required, opened a new log
I20260806 03:41:52.655121 32534 ts_tablet_manager.cc:1397] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: Time spent bootstrapping tablet: real 0.103s user 0.033s sys 0.055s
I20260806 03:41:52.655301 32206 leader_election.cc:302] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.656158 32595 raft_consensus.cc:2781] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.656538 32595 raft_consensus.cc:482] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.656919 32595 raft_consensus.cc:3037] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.657529 32534 raft_consensus.cc:348] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.658206 32534 raft_consensus.cc:374] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.658509 32534 raft_consensus.cc:729] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.654325 32536 ts_tablet_manager.cc:1428] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260806 03:41:52.659109 32534 consensus_queue.cc:260] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.660461 32536 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.661134 32534 ts_tablet_manager.cc:1428] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 03:41:52.662695 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.663324 32542 raft_consensus.cc:374] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.663770 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.663828 32542 raft_consensus.cc:729] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.664404 32479 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.665104 32542 consensus_queue.cc:260] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
W20260806 03:41:52.666924 32207 leader_election.cc:341] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.668696 32542 ts_tablet_manager.cc:1428] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.018s user 0.004s sys 0.003s
I20260806 03:41:52.670045 32542 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.672473 32479 raft_consensus.cc:2454] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.674400 32595 raft_consensus.cc:504] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.677103 32207 leader_election.cc:302] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters: d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.678583 32603 raft_consensus.cc:2781] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.678539 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.679329 32603 raft_consensus.cc:686] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.679649 32366 raft_consensus.cc:3037] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.680480 32603 consensus_queue.cc:237] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.686352 32602 leader_election.cc:288] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.692236 32548 raft_consensus.cc:482] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.693055 32548 raft_consensus.cc:504] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
W20260806 03:41:52.694201 32518 tablet.cc:2369] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:41:52.698899 32595 leader_election.cc:288] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.696594 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.697621 32548 leader_election.cc:288] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.701574 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
W20260806 03:41:52.704768 32319 leader_election.cc:341] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.705102 32253 raft_consensus.cc:2454] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 0.
I20260806 03:41:52.705271 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.705814 32366 raft_consensus.cc:2454] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
W20260806 03:41:52.706970 32207 leader_election.cc:341] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.707625 32206 leader_election.cc:302] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters: 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.708601 32603 raft_consensus.cc:2781] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.708969 32603 raft_consensus.cc:686] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.709494 32314 leader_election.cc:302] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters: 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.709751 32603 consensus_queue.cc:237] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.710670 32548 raft_consensus.cc:2781] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.711058 32548 raft_consensus.cc:482] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.711404 32548 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.714067 32536 tablet_bootstrap.cc:654] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.720356 32542 tablet_bootstrap.cc:654] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.721722 32101 catalog_manager.cc:5495] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.726217 32548 raft_consensus.cc:504] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.728681 32548 leader_election.cc:288] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.729578 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:52.730206 32253 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.732055 32542 tablet_bootstrap.cc:492] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.732513 32542 ts_tablet_manager.cc:1397] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.005s
I20260806 03:41:52.732951 32536 tablet_bootstrap.cc:492] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.733470 32536 ts_tablet_manager.cc:1397] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.073s user 0.018s sys 0.004s
I20260806 03:41:52.735435 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.735198 32542 raft_consensus.cc:348] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.736086 32542 raft_consensus.cc:374] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.736395 32542 raft_consensus.cc:729] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
W20260806 03:41:52.736795 32319 leader_election.cc:341] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.736294 32536 raft_consensus.cc:348] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.737414 32536 raft_consensus.cc:374] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.737855 32536 raft_consensus.cc:729] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.738531 32603 raft_consensus.cc:482] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.738608 32536 consensus_queue.cc:260] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.739043 32603 raft_consensus.cc:504] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.741057 32603 leader_election.cc:288] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.741518 32536 ts_tablet_manager.cc:1428] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260806 03:41:52.741856 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.742758 32536 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
W20260806 03:41:52.743299 32206 leader_election.cc:341] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.743206 32100 catalog_manager.cc:5495] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.744047 32253 raft_consensus.cc:2454] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 1.
I20260806 03:41:52.745316 32314 leader_election.cc:302] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters: 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.746565 32548 raft_consensus.cc:2781] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.747069 32548 raft_consensus.cc:686] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:52.747895 32548 consensus_queue.cc:237] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.747852 32100 catalog_manager.cc:5495] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.749229 32536 tablet_bootstrap.cc:654] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.737351 32542 consensus_queue.cc:260] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.750928 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
W20260806 03:41:52.752528 32207 leader_election.cc:341] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.753108 32542 ts_tablet_manager.cc:1428] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.020s user 0.006s sys 0.002s
I20260806 03:41:52.754136 32542 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.753139 32207 leader_election.cc:302] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966; no voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.755806 32603 raft_consensus.cc:2726] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.760620 32542 tablet_bootstrap.cc:654] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.767073 32603 raft_consensus.cc:482] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.767741 32603 raft_consensus.cc:504] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.771549 32603 leader_election.cc:288] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.772037 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
W20260806 03:41:52.773880 32206 leader_election.cc:341] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.775542 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.776468 32479 raft_consensus.cc:2454] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.778250 32207 leader_election.cc:302] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters: d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.779978 32603 raft_consensus.cc:2781] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.780550 32603 raft_consensus.cc:482] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.780968 32603 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.785470 32536 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.785979 32536 ts_tablet_manager.cc:1397] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.024s
I20260806 03:41:52.788166 32542 tablet_bootstrap.cc:492] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.788648 32542 ts_tablet_manager.cc:1397] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.010s
I20260806 03:41:52.788496 32603 raft_consensus.cc:504] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.788584 32536 raft_consensus.cc:348] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.789381 32536 raft_consensus.cc:374] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.789628 32536 raft_consensus.cc:729] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.788544 32101 catalog_manager.cc:5495] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.790237 32536 consensus_queue.cc:260] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.791373 32542 raft_consensus.cc:348] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.791548 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.792050 32542 raft_consensus.cc:374] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.792330 32542 raft_consensus.cc:729] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.792713 32536 ts_tablet_manager.cc:1428] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
W20260806 03:41:52.793043 32206 leader_election.cc:341] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:41:52.792980 32542 consensus_queue.cc:260] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.793571 32536 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.794216 32581 raft_consensus.cc:482] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.794649 32581 raft_consensus.cc:504] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.795044 32542 ts_tablet_manager.cc:1428] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 03:41:52.794991 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.795701 32479 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.796589 32581 leader_election.cc:288] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:52.797919 32542 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.799089 32536 tablet_bootstrap.cc:654] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.800716 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.801481 32253 raft_consensus.cc:2379] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89cc456269ab41c98e1b3f7813c490d8 in current term 1: Already voted for candidate d558039cc1914d328d0e3f9e637d66f1 in this term.
I20260806 03:41:52.803973 32542 tablet_bootstrap.cc:654] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.804742 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.806672 32429 leader_election.cc:302] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8; no voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.808311 32601 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.815146 32542 tablet_bootstrap.cc:492] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.815609 32542 ts_tablet_manager.cc:1397] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.002s
I20260806 03:41:52.818298 32603 leader_election.cc:288] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.818215 32542 raft_consensus.cc:348] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.819201 32542 raft_consensus.cc:374] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.819509 32542 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.820334 32542 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.818215 32479 raft_consensus.cc:2454] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.822002 32536 tablet_bootstrap.cc:492] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.822485 32536 ts_tablet_manager.cc:1397] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.020s
I20260806 03:41:52.823314 32542 ts_tablet_manager.cc:1428] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:41:52.823470 32601 raft_consensus.cc:2726] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.823905 32207 leader_election.cc:302] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters: d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:52.824244 32542 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: Bootstrap starting.
I20260806 03:41:52.825107 32603 raft_consensus.cc:2781] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.825618 32603 raft_consensus.cc:686] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.826596 32603 consensus_queue.cc:237] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.825587 32536 raft_consensus.cc:348] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.829012 32536 raft_consensus.cc:374] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.829329 32536 raft_consensus.cc:729] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.829850 32542 tablet_bootstrap.cc:654] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.830197 32536 consensus_queue.cc:260] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.832459 32536 ts_tablet_manager.cc:1428] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260806 03:41:52.833673 32536 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: Bootstrap starting.
I20260806 03:41:52.840911 32537 raft_consensus.cc:482] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.840999 32536 tablet_bootstrap.cc:654] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: Neither blocks nor log segments found. Creating new log.
I20260806 03:41:52.841485 32537 raft_consensus.cc:504] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.843490 32537 leader_election.cc:288] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.845141 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:52.845338 32548 consensus_queue.cc:1035] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:52.845645 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.846038 32366 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.846235 32479 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 0.
I20260806 03:41:52.847559 32206 leader_election.cc:302] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.848413 32537 raft_consensus.cc:2781] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.848748 32537 raft_consensus.cc:482] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.849080 32537 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.859848 32537 raft_consensus.cc:504] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.863224 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:52.863945 32366 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.864068 32542 tablet_bootstrap.cc:492] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: No bootstrap required, opened a new log
I20260806 03:41:52.864671 32542 ts_tablet_manager.cc:1397] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.002s
I20260806 03:41:52.869635 32542 raft_consensus.cc:348] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.870543 32542 raft_consensus.cc:374] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.870868 32542 raft_consensus.cc:729] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.871763 32536 tablet_bootstrap.cc:492] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: No bootstrap required, opened a new log
I20260806 03:41:52.877892 32536 ts_tablet_manager.cc:1397] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.007s
I20260806 03:41:52.871803 32576 consensus_queue.cc:1035] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260806 03:41:52.880362 32536 raft_consensus.cc:348] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.879853 32542 consensus_queue.cc:260] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.881098 32536 raft_consensus.cc:374] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:41:52.881423 32536 raft_consensus.cc:729] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Initialized, Role: FOLLOWER
I20260806 03:41:52.882050 32536 consensus_queue.cc:260] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.882827 32542 ts_tablet_manager.cc:1428] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260806 03:41:52.878646 32366 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.884877 32536 ts_tablet_manager.cc:1428] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:41:52.885192 32206 leader_election.cc:302] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.886135 32595 raft_consensus.cc:2781] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.886754 32595 raft_consensus.cc:686] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:52.887704 32595 consensus_queue.cc:237] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.894986 32537 leader_election.cc:288] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 1 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.895712 32101 catalog_manager.cc:5495] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.899789 32548 raft_consensus.cc:482] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.900234 32548 raft_consensus.cc:504] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.901903 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.903442 32548 leader_election.cc:288] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.903816 32576 raft_consensus.cc:482] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.902527 32478 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.904331 32576 raft_consensus.cc:504] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.905064 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.905665 32479 raft_consensus.cc:2379] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d558039cc1914d328d0e3f9e637d66f1 in current term 1: Already voted for candidate 0bc3e76ecad34049bbecde2881393966 in this term.
I20260806 03:41:52.908300 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.908903 32479 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 0.
I20260806 03:41:52.909977 32319 leader_election.cc:302] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.906395 32576 leader_election.cc:288] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.911015 32576 raft_consensus.cc:2781] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:41:52.911490 32576 raft_consensus.cc:482] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:41:52.911978 32576 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.913367 32478 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 1.
I20260806 03:41:52.916671 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.917660 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.918017 32314 leader_election.cc:302] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d558039cc1914d328d0e3f9e637d66f1; no voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.918286 32250 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 0.
I20260806 03:41:52.920094 32548 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.923967 32576 raft_consensus.cc:504] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.926033 32576 leader_election.cc:288] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.927155 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:52.927181 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:52.927898 32478 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.927951 32253 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:52.930037 32548 raft_consensus.cc:2726] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:52.931607 32101 catalog_manager.cc:5495] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 1 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.935050 32253 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 1.
I20260806 03:41:52.936233 32314 leader_election.cc:302] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:52.936740 32478 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 1.
I20260806 03:41:52.937163 32548 raft_consensus.cc:2781] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:41:52.938449 32548 raft_consensus.cc:686] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:52.939321 32548 consensus_queue.cc:237] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.947338 32101 catalog_manager.cc:5495] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:52.992204 32548 raft_consensus.cc:482] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:52.992726 32548 raft_consensus.cc:504] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:52.994455 32548 leader_election.cc:288] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:52.995303 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:52.995522 32478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89cc456269ab41c98e1b3f7813c490d8" is_pre_election: true
I20260806 03:41:52.996313 32478 raft_consensus.cc:2379] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d558039cc1914d328d0e3f9e637d66f1 in current term 1: Already voted for candidate 0bc3e76ecad34049bbecde2881393966 in this term.
I20260806 03:41:52.997144 32319 leader_election.cc:302] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d558039cc1914d328d0e3f9e637d66f1; no voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:52.997787 32548 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:53.002463 32548 raft_consensus.cc:2726] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:53.031283 32595 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.041733 32537 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.042727 32601 raft_consensus.cc:482] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:53.043215 32601 raft_consensus.cc:504] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.046175 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:53.048094 32601 leader_election.cc:288] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:53.049595 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1cfe35f4f645e4af266acefe4269e0" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:53.050479 32366 raft_consensus.cc:2379] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 89cc456269ab41c98e1b3f7813c490d8 in current term 1: Already voted for candidate 0bc3e76ecad34049bbecde2881393966 in this term.
I20260806 03:41:53.051424 32429 leader_election.cc:302] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8; no voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:53.052181 32601 raft_consensus.cc:3037] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:53.056984 32601 raft_consensus.cc:2726] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:53.073349 32537 consensus_queue.cc:1035] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.083501 32602 consensus_queue.cc:1035] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.106024 32366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:57214
I20260806 03:41:53.106755 32366 raft_consensus.cc:595] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:41:53.119802 32602 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.130579 32601 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:41:53.130752 32548 consensus_queue.cc:1035] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.131637 32601 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.132579 32478 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:41:53.141803 32581 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.143275 32548 consensus_queue.cc:1035] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.145073 32602 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.147425 32475 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
mode: GRACEFUL
new_leader_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:33096
I20260806 03:41:53.148049 32475 raft_consensus.cc:595] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 LEADER]: Received request to transfer leadership to TS d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:53.148927 32601 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:53.149603 32252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc3e76ecad34049bbecde2881393966" is_pre_election: true
I20260806 03:41:53.152245 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558039cc1914d328d0e3f9e637d66f1" is_pre_election: true
I20260806 03:41:53.154263 32429 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8; no voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:53.160846 32581 raft_consensus.cc:2726] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:41:53.171118 32601 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.171478 32252 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:58536
I20260806 03:41:53.172075 32252 raft_consensus.cc:595] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Received request to transfer leadership to TS d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:53.182221 32250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:58536
I20260806 03:41:53.182838 32250 raft_consensus.cc:595] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Received request to transfer leadership to TS d558039cc1914d328d0e3f9e637d66f1
I20260806 03:41:53.188632 32365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
mode: GRACEFUL
new_leader_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:57214
I20260806 03:41:53.189222 32365 raft_consensus.cc:595] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Received request to transfer leadership to TS 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:53.190963 32576 raft_consensus.cc:979] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: : Instructing follower 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:53.191408 32579 raft_consensus.cc:1067] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Signalling peer 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:53.192906 32479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:33092
I20260806 03:41:53.193840 32479 raft_consensus.cc:482] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:53.194216 32479 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.196238 32602 consensus_queue.cc:1035] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:53.200737 32479 raft_consensus.cc:504] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.202894 32479 leader_election.cc:288] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:53.203678 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:53.204039 32365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd12335f0f8e4005a88ac057dda964fc" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:53.204217 32250 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.204669 32365 raft_consensus.cc:3032] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:53.204998 32365 raft_consensus.cc:729] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:53.205849 32365 consensus_queue.cc:260] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.207363 32365 raft_consensus.cc:3037] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.211580 32537 consensus_queue.cc:1035] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.212560 32250 raft_consensus.cc:2454] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:53.213690 32425 leader_election.cc:302] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:53.214563 32601 raft_consensus.cc:2781] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:53.215019 32601 raft_consensus.cc:686] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:41:53.215837 32601 consensus_queue.cc:237] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.227658 32365 raft_consensus.cc:2454] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:53.228937 32101 catalog_manager.cc:5495] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 1 to 2, leader changed from d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66) to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:53.234079 32601 raft_consensus.cc:979] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: : Instructing follower d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.234603 32601 raft_consensus.cc:1067] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 LEADER]: Signalling peer d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.237030 32363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:57208
I20260806 03:41:53.237632 32363 raft_consensus.cc:482] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:53.237993 32363 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.244252 32363 raft_consensus.cc:504] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.246261 32363 leader_election.cc:288] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:53.246958 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:53.247295 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:53.247536 32250 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.247885 32474 raft_consensus.cc:3032] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:53.248142 32474 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:41:53.248824 32474 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.249743 32474 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.252362 32250 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.253360 32314 leader_election.cc:302] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:53.254068 32579 raft_consensus.cc:2781] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:53.254511 32579 raft_consensus.cc:686] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:53.254808 32474 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.255246 32579 consensus_queue.cc:237] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.261299 32102 catalog_manager.cc:5495] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 1 to 2, leader changed from 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67) to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:53.389372 32579 raft_consensus.cc:979] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:41:53.389886 32579 raft_consensus.cc:1067] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:41:53.391717 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58512
I20260806 03:41:53.392236 32250 raft_consensus.cc:482] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:53.392832 32250 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.397825 32250 raft_consensus.cc:504] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.399452 32250 leader_election.cc:288] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 2 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:53.400539 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:53.400559 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0eaba71e2dc24ac39581c7271f5c201e" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:53.401155 32474 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.401238 32363 raft_consensus.cc:3032] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:53.401583 32363 raft_consensus.cc:729] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:53.402279 32363 consensus_queue.cc:260] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.403294 32363 raft_consensus.cc:3037] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.406563 32474 raft_consensus.cc:2454] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 2.
I20260806 03:41:53.407673 32207 leader_election.cc:302] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:41:53.408391 32537 raft_consensus.cc:2781] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:53.408504 32363 raft_consensus.cc:2454] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 2.
I20260806 03:41:53.408921 32537 raft_consensus.cc:686] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 2 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:53.409849 32537 consensus_queue.cc:237] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.412779 32602 consensus_queue.cc:1035] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.415449 32537 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.417347 32102 catalog_manager.cc:5495] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 1 to 2, leader changed from d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 2 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:53.424235 32602 raft_consensus.cc:979] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: : Instructing follower d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.424810 32602 raft_consensus.cc:1067] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Signalling peer d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.429024 32587 raft_consensus.cc:979] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: : Instructing follower d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.429494 32587 consensus_queue.cc:1035] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.429955 32363 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:57192
I20260806 03:41:53.430620 32363 raft_consensus.cc:482] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:53.429524 32595 raft_consensus.cc:1067] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Signalling peer d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:41:53.430938 32363 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.433280 32366 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:57192
I20260806 03:41:53.433852 32366 raft_consensus.cc:482] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:53.434223 32366 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.440750 32587 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.441614 32366 raft_consensus.cc:504] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.444419 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:53.445086 32250 raft_consensus.cc:3032] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:53.445420 32250 raft_consensus.cc:729] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:53.446588 32250 consensus_queue.cc:260] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.447522 32250 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.450305 32363 raft_consensus.cc:504] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.451431 32365 raft_consensus.cc:1226] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Rejecting Update request from peer 0bc3e76ecad34049bbecde2881393966 for earlier term 1. Current term is 2. Ops: []
I20260806 03:41:53.454910 32366 leader_election.cc:288] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:53.455173 32587 consensus_queue.cc:1046] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.456166 32587 raft_consensus.cc:3032] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:53.456471 32587 raft_consensus.cc:729] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:53.456516 32363 leader_election.cc:288] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:41:53.457238 32587 consensus_queue.cc:260] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.458804 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:53.459583 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d446cdb829c1455799c42cf1d006f617" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:53.459863 32587 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.460171 32474 raft_consensus.cc:3037] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.461010 32250 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.462285 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:41:53.462869 32479 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:53.465621 32314 leader_election.cc:302] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:53.466480 32579 raft_consensus.cc:2781] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:53.466960 32579 raft_consensus.cc:686] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:53.467809 32579 consensus_queue.cc:237] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.468515 32474 raft_consensus.cc:2454] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.469573 32479 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.469705 32319 leader_election.cc:302] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:53.470665 32548 raft_consensus.cc:2781] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:53.471118 32548 raft_consensus.cc:686] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:41:53.471925 32548 consensus_queue.cc:237] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:53.477839 32101 catalog_manager.cc:5495] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 1 to 2, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:53.480393 32253 raft_consensus.cc:2454] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:41:53.489617 32100 catalog_manager.cc:5495] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 1 to 2, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:41:53.556578 32576 consensus_queue.cc:1035] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:41:53.561192 32576 consensus_queue.cc:1035] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:41:53.736747 32253 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.738093 32548 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.745546 32479 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.747315 32579 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.754422 32363 raft_consensus.cc:1261] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.756229 32655 consensus_queue.cc:1035] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.758136 32253 raft_consensus.cc:1261] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.759462 32663 consensus_queue.cc:1035] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:53.840946 32479 raft_consensus.cc:1261] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.842139 32587 consensus_queue.cc:1035] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.849387 32363 raft_consensus.cc:1261] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.851166 32595 consensus_queue.cc:1035] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.868768 32479 raft_consensus.cc:1261] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.869869 32579 consensus_queue.cc:1035] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.877718 32253 raft_consensus.cc:1261] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.879566 32579 consensus_queue.cc:1035] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.908776 32253 raft_consensus.cc:1261] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.909912 32579 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:53.918442 32479 raft_consensus.cc:1261] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:53.919780 32576 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:56.219847 31273 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 03:41:56.220618 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:56.241665 32253 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:41:56.242182 32253 raft_consensus.cc:595] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Received request to transfer leadership to TS 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:56.245163 32253 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:41:56.246179 32253 raft_consensus.cc:595] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Received request to transfer leadership to TS 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:41:56.248153 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:41:56.248637 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:56.372221 32676 raft_consensus.cc:979] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: : Instructing follower 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:56.372730 32676 raft_consensus.cc:1067] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Signalling peer 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:56.374266 32479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:33090
I20260806 03:41:56.375006 32479 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:56.375311 32479 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.380702 32479 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.382247 32479 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:56.383297 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:56.383445 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:56.383908 32253 raft_consensus.cc:3032] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:56.384020 32363 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.384485 32253 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:56.385332 32253 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.386302 32253 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.389443 32363 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:56.391063 32253 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:56.391050 32429 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:56.392167 32669 raft_consensus.cc:2781] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:56.393272 32669 raft_consensus.cc:686] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:41:56.394110 32669 consensus_queue.cc:237] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.402452 32100 catalog_manager.cc:5495] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 1 to 2, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:56.406927 32676 raft_consensus.cc:979] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: : Instructing follower 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:56.407438 32670 raft_consensus.cc:1067] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Signalling peer 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:41:56.409193 32479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:33090
I20260806 03:41:56.409703 32479 raft_consensus.cc:482] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:41:56.410178 32479 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.418434 32479 raft_consensus.cc:504] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.420080 32479 leader_election.cc:288] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:41:56.420893 32253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:41:56.421159 32363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:41:56.421543 32253 raft_consensus.cc:3032] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:41:56.421701 32363 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.421861 32253 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:41:56.422716 32253 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.423817 32253 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:41:56.428905 32363 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:56.429639 32253 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 2.
I20260806 03:41:56.430351 32429 leader_election.cc:302] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:41:56.431257 32669 raft_consensus.cc:2781] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:41:56.431725 32669 raft_consensus.cc:686] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:41:56.432446 32669 consensus_queue.cc:237] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:41:56.441273 32100 catalog_manager.cc:5495] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 1 to 2, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:41:56.861166 32363 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:56.862747 32669 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:56.902221 32253 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:56.906268 32669 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:41:56.922370 32253 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:56.923837 32669 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:56.934405 32363 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:41:56.936190 32669 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:41:57.249481 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:57.256731 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:57.257212 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:58.258189 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:58.265148 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:58.265642 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:41:59.266462 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:41:59.272388 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:41:59.272859 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:00.274012 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:00.281280 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:00.281709 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:01.282750 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:01.288323 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:01.288715 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:02.289495 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:02.295295 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:02.295653 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:03.296531 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:03.301713 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:03.302165 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:04.303247 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:04.309669 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:04.310194 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:05.311168 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:05.316864 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:05.317375 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:06.346591 31273 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 03:42:06.353540 32479 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:33096
I20260806 03:42:06.354247 32479 raft_consensus.cc:595] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:42:06.359347 32479 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:33096
I20260806 03:42:06.359792 32479 raft_consensus.cc:595] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:42:06.370157 32366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:57214
I20260806 03:42:06.370752 32366 raft_consensus.cc:595] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:42:06.375526 32366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:57214
I20260806 03:42:06.376121 32366 raft_consensus.cc:595] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:42:06.381055 32366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
mode: GRACEFUL
new_leader_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:57214
I20260806 03:42:06.381474 32366 raft_consensus.cc:595] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Received request to transfer leadership to TS 0bc3e76ecad34049bbecde2881393966
I20260806 03:42:06.497277 32718 raft_consensus.cc:979] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.497749 32718 raft_consensus.cc:1067] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.499342 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58512
I20260806 03:42:06.499847 32250 raft_consensus.cc:482] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:06.500214 32250 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.507771 32250 raft_consensus.cc:504] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.509769 32669 raft_consensus.cc:979] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.510247 32669 raft_consensus.cc:1067] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.510553 32250 leader_election.cc:288] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:06.510926 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:06.511197 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:06.511796 32479 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.511760 32366 raft_consensus.cc:3032] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:42:06.512334 32366 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:42:06.512514 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58528
I20260806 03:42:06.513088 32250 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:06.513037 32366 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.513533 32250 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.514300 32366 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.519273 32479 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.520370 32366 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.520687 32207 leader_election.cc:302] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:42:06.520514 32250 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.521689 32717 raft_consensus.cc:2781] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:42:06.522166 32717 raft_consensus.cc:686] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:06.523620 32250 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:06.523070 32717 consensus_queue.cc:237] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.524806 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:06.525445 32474 raft_consensus.cc:3032] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:42:06.525771 32474 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:42:06.526435 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:06.527099 32366 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.526726 32474 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.528717 32474 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.533231 32102 catalog_manager.cc:5495] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 2 to 3, leader changed from d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:06.534762 32366 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.536134 32206 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:06.537004 32717 raft_consensus.cc:2781] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:42:06.537177 32474 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.537698 32717 raft_consensus.cc:686] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:06.538990 32717 consensus_queue.cc:237] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.558091 32102 catalog_manager.cc:5495] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 2 to 3, leader changed from 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:06.639346 32669 raft_consensus.cc:979] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.639973 32721 raft_consensus.cc:1067] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.641597 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58528
I20260806 03:42:06.642087 32250 raft_consensus.cc:482] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:06.642437 32250 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.647496 32250 raft_consensus.cc:504] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.649242 32250 leader_election.cc:288] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:06.650059 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:06.650282 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:06.650727 32366 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.651132 32474 raft_consensus.cc:3032] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:42:06.651407 32474 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:42:06.652072 32474 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.653040 32474 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.655932 32366 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.657116 32206 leader_election.cc:302] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:06.657913 32717 raft_consensus.cc:2781] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:42:06.657996 32474 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.658556 32717 raft_consensus.cc:686] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:06.659303 32717 consensus_queue.cc:237] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.666231 32102 catalog_manager.cc:5495] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 2 to 3, leader changed from 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:06.685518 32719 raft_consensus.cc:979] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.686043 32718 raft_consensus.cc:1067] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.687458 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58512
I20260806 03:42:06.687978 32250 raft_consensus.cc:482] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:06.688262 32250 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.693185 32250 raft_consensus.cc:504] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.694904 32250 leader_election.cc:288] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:06.695976 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:06.695976 32474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f0c226c59e6412cbfc08cf7b7e7c577" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:06.696770 32366 raft_consensus.cc:3032] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:42:06.696805 32474 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.697243 32366 raft_consensus.cc:729] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:42:06.698088 32366 consensus_queue.cc:260] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.699379 32366 raft_consensus.cc:3037] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:42:06.703945 32474 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.705070 32207 leader_election.cc:302] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:42:06.705952 32717 raft_consensus.cc:2781] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:42:06.706054 32366 raft_consensus.cc:2454] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 3.
I20260806 03:42:06.706553 32717 raft_consensus.cc:686] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:06.707564 32717 consensus_queue.cc:237] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.715243 32102 catalog_manager.cc:5495] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 2 to 3, leader changed from d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 3 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:06.818789 32719 raft_consensus.cc:979] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: : Instructing follower 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.819314 32718 raft_consensus.cc:1067] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Signalling peer 0bc3e76ecad34049bbecde2881393966 to start an election
I20260806 03:42:06.820765 32250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
from {username='slave'} at 127.0.0.1:58512
I20260806 03:42:06.821249 32250 raft_consensus.cc:482] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:06.821517 32250 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:06.827451 32250 raft_consensus.cc:504] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.828928 32250 leader_election.cc:288] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 2 election: Requested vote from peers d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:06.829890 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:06.830006 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e10bdb723a5f47daa1fb1990ffb2b4b1" candidate_uuid: "0bc3e76ecad34049bbecde2881393966" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:06.830585 32479 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:06.830529 32366 raft_consensus.cc:3032] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:06.831074 32366 raft_consensus.cc:729] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:42:06.831807 32366 consensus_queue.cc:260] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.832888 32366 raft_consensus.cc:3037] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:06.835354 32479 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 2.
I20260806 03:42:06.836342 32207 leader_election.cc:302] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc3e76ecad34049bbecde2881393966, 89cc456269ab41c98e1b3f7813c490d8; no voters:
I20260806 03:42:06.837121 32717 raft_consensus.cc:2781] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:06.837587 32717 raft_consensus.cc:686] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 2 LEADER]: Becoming Leader. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:06.838294 32366 raft_consensus.cc:2454] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bc3e76ecad34049bbecde2881393966 in term 2.
I20260806 03:42:06.838495 32717 consensus_queue.cc:237] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:06.845276 32102 catalog_manager.cc:5495] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 reported cstate change: term changed from 1 to 2, leader changed from d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66) to 0bc3e76ecad34049bbecde2881393966 (127.30.138.65). New cstate: current_term: 2 leader_uuid: "0bc3e76ecad34049bbecde2881393966" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:06.983461 32366 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:06.984804 32717 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:06.996204 32479 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:06.997763 32670 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:07.029026 32366 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.030392 32670 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:07.041193 32479 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.042480 32717 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:07.069682 32479 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.070959 32717 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:07.080036 32366 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.081578 32717 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:07.126765 32479 raft_consensus.cc:1261] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.127957 32717 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:07.136606 32366 raft_consensus.cc:1261] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:42:07.139259 32745 consensus_queue.cc:1035] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:07.237866 32479 raft_consensus.cc:1261] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:07.239189 32717 consensus_queue.cc:1035] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:07.239948 32366 raft_consensus.cc:1261] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Refusing update from remote peer 0bc3e76ecad34049bbecde2881393966: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:07.241106 32717 consensus_queue.cc:1035] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:09.419960 31273 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 03:42:09.420830 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:09.430162 32250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:42:09.430820 32250 raft_consensus.cc:595] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Received request to transfer leadership to TS d558039cc1914d328d0e3f9e637d66f1
I20260806 03:42:09.433173 32250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:42:09.433650 32250 raft_consensus.cc:595] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Received request to transfer leadership to TS 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:42:09.435748 32250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:42:09.436200 32250 raft_consensus.cc:595] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Received request to transfer leadership to TS 89cc456269ab41c98e1b3f7813c490d8
I20260806 03:42:09.438313 32250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "0bc3e76ecad34049bbecde2881393966"
mode: GRACEFUL
new_leader_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:58544
I20260806 03:42:09.438768 32250 raft_consensus.cc:595] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Received request to transfer leadership to TS d558039cc1914d328d0e3f9e637d66f1
I20260806 03:42:09.439646 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:42:09.440127 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:09.498059 32717 raft_consensus.cc:979] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: : Instructing follower d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:42:09.498617 32766 raft_consensus.cc:1067] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Signalling peer d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:42:09.499958 32366 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:57192
I20260806 03:42:09.500489 32366 raft_consensus.cc:482] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:09.500774 32366 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.505175 32366 raft_consensus.cc:504] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.507310 32366 leader_election.cc:288] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 4 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:09.508212 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:42:09.508381 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb542237f30b4f9282429bc416260fa7" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:09.508765 32250 raft_consensus.cc:3032] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:42:09.508914 32479 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.509045 32250 raft_consensus.cc:729] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:09.509752 32250 consensus_queue.cc:260] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.510756 32250 raft_consensus.cc:3037] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.513669 32479 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 4.
I20260806 03:42:09.514899 32319 leader_election.cc:302] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:09.515273 32250 raft_consensus.cc:2454] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 4.
I20260806 03:42:09.515584 32765 raft_consensus.cc:2781] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:42:09.516790 32765 raft_consensus.cc:686] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 4 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:42:09.517583 32765 consensus_queue.cc:237] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.524983 32766 raft_consensus.cc:979] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: : Instructing follower 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:42:09.524235 32102 catalog_manager.cc:5495] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 3 to 4, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:09.526491 32766 raft_consensus.cc:1067] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Signalling peer 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:42:09.529126 32479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:33090
I20260806 03:42:09.529776 32479 raft_consensus.cc:482] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:09.530189 32479 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.538229 32479 raft_consensus.cc:504] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.540802 32479 leader_election.cc:288] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 4 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:42:09.541669 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:42:09.541838 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e822611997a47e8bc60af35416dfddf" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:09.542256 32250 raft_consensus.cc:3032] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:42:09.542335 32366 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.542747 32250 raft_consensus.cc:729] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:09.543504 32250 consensus_queue.cc:260] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.544304 32250 raft_consensus.cc:3037] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.549052 32366 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 4.
I20260806 03:42:09.550084 32429 leader_election.cc:302] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:09.550846 32250 raft_consensus.cc:2454] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 4.
I20260806 03:42:09.550841 32721 raft_consensus.cc:2781] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:42:09.552449 32721 raft_consensus.cc:686] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 4 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:42:09.553215 32721 consensus_queue.cc:237] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.559504 32102 catalog_manager.cc:5495] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 3 to 4, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 4 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:09.673846 32717 raft_consensus.cc:979] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: : Instructing follower d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:42:09.674752 301 raft_consensus.cc:1067] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Signalling peer d558039cc1914d328d0e3f9e637d66f1 to start an election
I20260806 03:42:09.677068 32366 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e"
dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
from {username='slave'} at 127.0.0.1:57192
I20260806 03:42:09.677829 32366 raft_consensus.cc:482] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:09.678383 32366 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:09.686039 32366 raft_consensus.cc:504] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.687747 32366 leader_election.cc:288] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67:40459)
I20260806 03:42:09.689700 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:42:09.689898 32479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f813cc1db34443f9e7525785bfebd1e" candidate_uuid: "d558039cc1914d328d0e3f9e637d66f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
I20260806 03:42:09.690512 32250 raft_consensus.cc:3032] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:09.690629 32479 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:09.690886 32250 raft_consensus.cc:729] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:09.691802 32250 consensus_queue.cc:260] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.693611 32250 raft_consensus.cc:3037] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:09.699878 32479 raft_consensus.cc:2454] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:42:09.701681 32319 leader_election.cc:302] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:09.702576 32250 raft_consensus.cc:2454] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558039cc1914d328d0e3f9e637d66f1 in term 2.
I20260806 03:42:09.702975 32765 raft_consensus.cc:2781] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:09.703697 32765 raft_consensus.cc:686] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Becoming Leader. State: Replica: d558039cc1914d328d0e3f9e637d66f1, State: Running, Role: LEADER
I20260806 03:42:09.705076 32765 consensus_queue.cc:237] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.715257 32102 catalog_manager.cc:5495] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 reported cstate change: term changed from 1 to 2, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d558039cc1914d328d0e3f9e637d66f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:09.940584 301 raft_consensus.cc:979] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: : Instructing follower 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:42:09.941123 32717 raft_consensus.cc:1067] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Signalling peer 89cc456269ab41c98e1b3f7813c490d8 to start an election
I20260806 03:42:09.942803 32479 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86"
dest_uuid: "89cc456269ab41c98e1b3f7813c490d8"
from {username='slave'} at 127.0.0.1:33090
I20260806 03:42:09.943337 32479 raft_consensus.cc:482] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:09.943679 32479 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.948410 32479 raft_consensus.cc:504] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.950066 32479 leader_election.cc:288] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 4 election: Requested vote from peers 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911), d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973)
I20260806 03:42:09.951006 32250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0bc3e76ecad34049bbecde2881393966"
I20260806 03:42:09.951177 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47b707c5ab64b28abac16902d9fff86" candidate_uuid: "89cc456269ab41c98e1b3f7813c490d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d558039cc1914d328d0e3f9e637d66f1"
I20260806 03:42:09.951578 32250 raft_consensus.cc:3032] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:42:09.951820 32366 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.951906 32250 raft_consensus.cc:729] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0bc3e76ecad34049bbecde2881393966, State: Running, Role: LEADER
I20260806 03:42:09.954689 32250 consensus_queue.cc:260] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.955899 32250 raft_consensus.cc:3037] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:42:09.958348 32366 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 4.
I20260806 03:42:09.959416 32429 leader_election.cc:302] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cc456269ab41c98e1b3f7813c490d8, d558039cc1914d328d0e3f9e637d66f1; no voters:
I20260806 03:42:09.960232 32721 raft_consensus.cc:2781] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:42:09.960685 32721 raft_consensus.cc:686] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 LEADER]: Becoming Leader. State: Replica: 89cc456269ab41c98e1b3f7813c490d8, State: Running, Role: LEADER
I20260806 03:42:09.960932 32250 raft_consensus.cc:2454] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cc456269ab41c98e1b3f7813c490d8 in term 4.
I20260806 03:42:09.961805 32721 consensus_queue.cc:237] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } }
I20260806 03:42:09.972283 32102 catalog_manager.cc:5495] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 reported cstate change: term changed from 3 to 4, leader changed from 0bc3e76ecad34049bbecde2881393966 (127.30.138.65) to 89cc456269ab41c98e1b3f7813c490d8 (127.30.138.67). New cstate: current_term: 4 leader_uuid: "89cc456269ab41c98e1b3f7813c490d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:09.975641 32250 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:09.978106 32721 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:09.994247 32366 raft_consensus.cc:1261] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:09.997427 32721 consensus_queue.cc:1035] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:10.041684 32479 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:10.042937 32765 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:10.052055 32250 raft_consensus.cc:1261] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:10.053835 32767 consensus_queue.cc:1035] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:10.145244 32479 raft_consensus.cc:1261] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:10.146559 32767 consensus_queue.cc:1035] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89cc456269ab41c98e1b3f7813c490d8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:10.155879 32250 raft_consensus.cc:1261] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Refusing update from remote peer d558039cc1914d328d0e3f9e637d66f1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:10.157505 333 consensus_queue.cc:1035] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:10.393782 32250 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:10.395040 32721 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc3e76ecad34049bbecde2881393966" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 38911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:10.404110 32366 raft_consensus.cc:1261] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Refusing update from remote peer 89cc456269ab41c98e1b3f7813c490d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:42:10.405642 311 consensus_queue.cc:1035] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558039cc1914d328d0e3f9e637d66f1" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 39973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:10.441133 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:10.447243 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:10.447675 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:11.448591 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:11.455152 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:11.455686 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:12.456629 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:12.461867 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:12.462235 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:13.463125 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:13.468273 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:13.468663 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:14.469465 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:14.474625 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:14.475056 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:15.475894 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:15.481516 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:15.481912 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:16.482802 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:16.488350 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:16.488811 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:17.489674 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:17.494964 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:17.495394 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:18.496344 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:18.502208 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:42:18.502671 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:19.532958 31273 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 03:42:19.533497 31273 tablet_server.cc:178] TabletServer@127.30.138.65:0 shutting down...
I20260806 03:42:19.565733 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:42:19.566490 31273 tablet_replica.cc:330] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.567147 31273 raft_consensus.cc:2229] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.567677 31273 raft_consensus.cc:2258] T d446cdb829c1455799c42cf1d006f617 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.569775 31273 tablet_replica.cc:330] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.570427 31273 raft_consensus.cc:2229] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.571218 31273 raft_consensus.cc:2258] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 0bc3e76ecad34049bbecde2881393966 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.572933 31273 tablet_replica.cc:330] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.573457 31273 raft_consensus.cc:2229] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.574250 31273 raft_consensus.cc:2258] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.576061 31273 tablet_replica.cc:330] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.576474 31273 raft_consensus.cc:2229] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.576828 31273 raft_consensus.cc:2258] T 3f813cc1db34443f9e7525785bfebd1e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.578333 31273 tablet_replica.cc:330] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.578918 31273 raft_consensus.cc:2229] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.579661 31273 raft_consensus.cc:2258] T eb1cfe35f4f645e4af266acefe4269e0 P 0bc3e76ecad34049bbecde2881393966 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.581805 31273 tablet_replica.cc:330] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.582309 31273 raft_consensus.cc:2229] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.583483 31273 raft_consensus.cc:2258] T 0eaba71e2dc24ac39581c7271f5c201e P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.585584 31273 tablet_replica.cc:330] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.586107 31273 raft_consensus.cc:2229] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.586560 31273 raft_consensus.cc:2258] T cd12335f0f8e4005a88ac057dda964fc P 0bc3e76ecad34049bbecde2881393966 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.588446 31273 tablet_replica.cc:330] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.588974 31273 raft_consensus.cc:2229] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.589385 31273 raft_consensus.cc:2258] T 7e822611997a47e8bc60af35416dfddf P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.591105 31273 tablet_replica.cc:330] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.591670 31273 raft_consensus.cc:2229] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.592070 31273 raft_consensus.cc:2258] T eb542237f30b4f9282429bc416260fa7 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.593546 31273 tablet_replica.cc:330] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966: stopping tablet replica
I20260806 03:42:19.594025 31273 raft_consensus.cc:2229] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.594436 31273 raft_consensus.cc:2258] T e47b707c5ab64b28abac16902d9fff86 P 0bc3e76ecad34049bbecde2881393966 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.624944 31273 tablet_server.cc:194] TabletServer@127.30.138.65:0 shutdown complete.
I20260806 03:42:19.644748 31273 tablet_server.cc:178] TabletServer@127.30.138.66:0 shutting down...
W20260806 03:42:19.664526 32425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.138.65:38911: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20260806 03:42:19.670128 32425 consensus_peers.cc:487] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 -> Peer 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911): Couldn't send request to peer 0bc3e76ecad34049bbecde2881393966. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:38911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:19.698242 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:42:19.699790 31273 tablet_replica.cc:330] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
W20260806 03:42:19.702286 32425 consensus_peers.cc:487] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 -> Peer 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911): Couldn't send request to peer 0bc3e76ecad34049bbecde2881393966. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:38911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:19.702214 31273 raft_consensus.cc:2229] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.705226 31273 raft_consensus.cc:2258] T 3f813cc1db34443f9e7525785bfebd1e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.709427 31273 tablet_replica.cc:330] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.710682 31273 raft_consensus.cc:2229] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 LEADER]: Raft consensus shutting down.
W20260806 03:42:19.710793 32314 consensus_peers.cc:487] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 -> Peer 0bc3e76ecad34049bbecde2881393966 (127.30.138.65:38911): Couldn't send request to peer 0bc3e76ecad34049bbecde2881393966. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.65:38911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:19.712127 31273 raft_consensus.cc:2258] T d446cdb829c1455799c42cf1d006f617 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.714082 31273 tablet_replica.cc:330] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.714738 31273 raft_consensus.cc:2229] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.715304 31273 raft_consensus.cc:2258] T e10bdb723a5f47daa1fb1990ffb2b4b1 P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.718143 31273 tablet_replica.cc:330] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.718829 31273 raft_consensus.cc:2229] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.719393 31273 raft_consensus.cc:2258] T 3f0c226c59e6412cbfc08cf7b7e7c577 P d558039cc1914d328d0e3f9e637d66f1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.721910 31273 tablet_replica.cc:330] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.722838 31273 raft_consensus.cc:2229] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.723642 31273 raft_consensus.cc:2258] T eb1cfe35f4f645e4af266acefe4269e0 P d558039cc1914d328d0e3f9e637d66f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.726666 31273 tablet_replica.cc:330] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.727419 31273 raft_consensus.cc:2229] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.728106 31273 raft_consensus.cc:2258] T e47b707c5ab64b28abac16902d9fff86 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.729949 31273 tablet_replica.cc:330] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.730579 31273 raft_consensus.cc:2229] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.731889 31273 raft_consensus.cc:2258] T eb542237f30b4f9282429bc416260fa7 P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.733742 31273 tablet_replica.cc:330] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.734231 31273 raft_consensus.cc:2229] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.734843 31273 raft_consensus.cc:2258] T cd12335f0f8e4005a88ac057dda964fc P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.736819 31273 tablet_replica.cc:330] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.737376 31273 raft_consensus.cc:2229] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.737880 31273 raft_consensus.cc:2258] T 0eaba71e2dc24ac39581c7271f5c201e P d558039cc1914d328d0e3f9e637d66f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.739806 31273 tablet_replica.cc:330] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1: stopping tablet replica
I20260806 03:42:19.740275 31273 raft_consensus.cc:2229] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.740796 31273 raft_consensus.cc:2258] T 7e822611997a47e8bc60af35416dfddf P d558039cc1914d328d0e3f9e637d66f1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.772153 31273 tablet_server.cc:194] TabletServer@127.30.138.66:0 shutdown complete.
I20260806 03:42:19.793857 31273 tablet_server.cc:178] TabletServer@127.30.138.67:0 shutting down...
I20260806 03:42:19.822768 31273 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:42:19.823717 31273 tablet_replica.cc:330] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.824422 31273 raft_consensus.cc:2229] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.824983 31273 raft_consensus.cc:2258] T eb1cfe35f4f645e4af266acefe4269e0 P 89cc456269ab41c98e1b3f7813c490d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.828220 31273 tablet_replica.cc:330] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.829334 31273 raft_consensus.cc:2229] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.830125 31273 raft_consensus.cc:2258] T e10bdb723a5f47daa1fb1990ffb2b4b1 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 03:42:19.830458 32429 consensus_peers.cc:487] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 -> Peer d558039cc1914d328d0e3f9e637d66f1 (127.30.138.66:39973): Couldn't send request to peer d558039cc1914d328d0e3f9e637d66f1. Status: Network error: Client connection negotiation failed: client connection to 127.30.138.66:39973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:19.832667 31273 tablet_replica.cc:330] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.833504 31273 raft_consensus.cc:2229] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.834059 31273 raft_consensus.cc:2258] T 3f0c226c59e6412cbfc08cf7b7e7c577 P 89cc456269ab41c98e1b3f7813c490d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.836017 31273 tablet_replica.cc:330] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.836500 31273 raft_consensus.cc:2229] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.837005 31273 raft_consensus.cc:2258] T 3f813cc1db34443f9e7525785bfebd1e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.838863 31273 tablet_replica.cc:330] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.839381 31273 raft_consensus.cc:2229] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.839808 31273 raft_consensus.cc:2258] T d446cdb829c1455799c42cf1d006f617 P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.841627 31273 tablet_replica.cc:330] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.842133 31273 raft_consensus.cc:2229] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.843335 31273 raft_consensus.cc:2258] T e47b707c5ab64b28abac16902d9fff86 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.845196 31273 tablet_replica.cc:330] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.845652 31273 raft_consensus.cc:2229] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.846191 31273 raft_consensus.cc:2258] T eb542237f30b4f9282429bc416260fa7 P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.848004 31273 tablet_replica.cc:330] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.848435 31273 raft_consensus.cc:2229] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.849352 31273 raft_consensus.cc:2258] T 7e822611997a47e8bc60af35416dfddf P 89cc456269ab41c98e1b3f7813c490d8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.851512 31273 tablet_replica.cc:330] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.851969 31273 raft_consensus.cc:2229] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.853057 31273 raft_consensus.cc:2258] T cd12335f0f8e4005a88ac057dda964fc P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.855191 31273 tablet_replica.cc:330] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8: stopping tablet replica
I20260806 03:42:19.855777 31273 raft_consensus.cc:2229] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:19.856324 31273 raft_consensus.cc:2258] T 0eaba71e2dc24ac39581c7271f5c201e P 89cc456269ab41c98e1b3f7813c490d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.888051 31273 tablet_server.cc:194] TabletServer@127.30.138.67:0 shutdown complete.
I20260806 03:42:19.909020 31273 master.cc:537] Master@127.30.138.126:44755 shutting down...
I20260806 03:42:19.933091 31273 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:42:19.933739 31273 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:19.934042 31273 tablet_replica.cc:330] T 00000000000000000000000000000000 P 03227a8f28c24f50aa93577c96b6e773: stopping tablet replica
I20260806 03:42:19.953040 31273 master.cc:552] Master@127.30.138.126:44755 shutdown complete.
0bc3e76ecad34049bbecde2881393966 5
89cc456269ab41c98e1b3f7813c490d8 1
d558039cc1914d328d0e3f9e637d66f1 4
0bc3e76ecad34049bbecde2881393966 3
89cc456269ab41c98e1b3f7813c490d8 3
d558039cc1914d328d0e3f9e637d66f1 4
0bc3e76ecad34049bbecde2881393966 8
89cc456269ab41c98e1b3f7813c490d8 1
d558039cc1914d328d0e3f9e637d66f1 1
0bc3e76ecad34049bbecde2881393966 4
89cc456269ab41c98e1b3f7813c490d8 3
d558039cc1914d328d0e3f9e637d66f1 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28819 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 03:42:19.996268 31273 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.30.138.126:46149
I20260806 03:42:19.997385 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:20.003871 362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.003844 361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.004575 365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:20.005262 31273 server_base.cc:1034] running on GCE node
I20260806 03:42:20.006474 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:42:20.006753 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:42:20.006925 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987740006910 us; error 0 us; skew 500 ppm
I20260806 03:42:20.007426 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:42:20.010171 31273 webserver.cc:460] Webserver started at http://127.30.138.126:44697/ using document root <none> and password file <none>
I20260806 03:42:20.010762 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:20.010985 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:20.011286 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:20.012655 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/master-0-root/instance:
uuid: "eed80bed2e2c4b49a8999c7f5e713c34"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.017448 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 03:42:20.021443 374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.022274 31273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 03:42:20.022684 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/master-0-root
uuid: "eed80bed2e2c4b49a8999c7f5e713c34"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.022991 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:20.057896 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:20.059331 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:20.099660 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.126:46149
I20260806 03:42:20.099752 450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.126:46149 every 8 connection(s)
I20260806 03:42:20.103816 452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.114952 452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Bootstrap starting.
I20260806 03:42:20.119601 452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:20.124310 452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: No bootstrap required, opened a new log
I20260806 03:42:20.126695 452 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER }
I20260806 03:42:20.127213 452 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:20.127506 452 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed80bed2e2c4b49a8999c7f5e713c34, State: Initialized, Role: FOLLOWER
I20260806 03:42:20.128177 452 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER }
I20260806 03:42:20.128664 452 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:42:20.128917 452 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:42:20.129208 452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:20.133600 452 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER }
I20260806 03:42:20.134193 452 leader_election.cc:302] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed80bed2e2c4b49a8999c7f5e713c34; no voters:
I20260806 03:42:20.135476 452 leader_election.cc:288] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:42:20.135929 456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:20.137315 456 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [term 1 LEADER]: Becoming Leader. State: Replica: eed80bed2e2c4b49a8999c7f5e713c34, State: Running, Role: LEADER
I20260806 03:42:20.138027 456 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER }
I20260806 03:42:20.138490 452 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:42:20.144762 457 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER } }
I20260806 03:42:20.144979 458 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed80bed2e2c4b49a8999c7f5e713c34. Latest consensus state: current_term: 1 leader_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed80bed2e2c4b49a8999c7f5e713c34" member_type: VOTER } }
I20260806 03:42:20.145787 457 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:20.145864 458 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34 [sys.catalog]: This master's current role is: LEADER
I20260806 03:42:20.152627 470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:42:20.157559 470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:42:20.160482 31273 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:42:20.167235 470 catalog_manager.cc:1347] Generated new cluster ID: deb8487da5b54769bf1fd769edcaeca2
I20260806 03:42:20.167610 470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:42:20.204007 470 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:42:20.205755 470 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:42:20.225706 470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Generated new TSK 0
I20260806 03:42:20.226536 470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:42:20.231684 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:20.237833 485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.239434 486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.241144 488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:20.241803 31273 server_base.cc:1034] running on GCE node
I20260806 03:42:20.242785 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:42:20.242978 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:42:20.243090 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987740243082 us; error 0 us; skew 500 ppm
I20260806 03:42:20.243597 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:42:20.246125 31273 webserver.cc:460] Webserver started at http://127.30.138.65:35865/ using document root <none> and password file <none>
I20260806 03:42:20.246762 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:20.246975 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:20.247220 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:20.248304 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/instance:
uuid: "ada5ef6656cb4420baf77a4b4342909e"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.253489 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 03:42:20.258164 495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.259335 31273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:42:20.259714 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root
uuid: "ada5ef6656cb4420baf77a4b4342909e"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.260013 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:20.283279 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:20.284547 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:20.286063 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:20.288383 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:20.288595 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.288851 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:20.289033 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.331835 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.65:45761
I20260806 03:42:20.331936 594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.65:45761 every 8 connection(s)
I20260806 03:42:20.336392 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:20.343631 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.345566 601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:20.352212 31273 server_base.cc:1034] running on GCE node
W20260806 03:42:20.352691 604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:20.353929 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:42:20.354189 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:42:20.354322 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987740354314 us; error 0 us; skew 500 ppm
I20260806 03:42:20.355038 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:42:20.355103 595 heartbeater.cc:347] Connected to a master server at 127.30.138.126:46149
I20260806 03:42:20.355641 595 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:20.356557 595 heartbeater.cc:511] Master 127.30.138.126:46149 requested a full tablet report, sending...
I20260806 03:42:20.357618 31273 webserver.cc:460] Webserver started at http://127.30.138.66:35307/ using document root <none> and password file <none>
I20260806 03:42:20.358168 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:20.358595 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:20.358948 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:20.359205 403 ts_manager.cc:194] Registered new tserver with Master: ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:20.360239 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/instance:
uuid: "d8469561a52c4bbc81dce786f1211ee6"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.361373 403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35370
I20260806 03:42:20.365191 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 03:42:20.369343 613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.370165 31273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 03:42:20.370556 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root
uuid: "d8469561a52c4bbc81dce786f1211ee6"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.370872 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:20.393374 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:20.394721 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:20.396306 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:20.398725 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:20.398948 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.399199 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:20.399375 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.445600 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.66:41639
I20260806 03:42:20.445729 711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.66:41639 every 8 connection(s)
I20260806 03:42:20.451180 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:20.459911 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.462518 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:20.465758 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:20.465768 31273 server_base.cc:1034] running on GCE node
I20260806 03:42:20.467535 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:42:20.467789 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:42:20.468022 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987740467999 us; error 0 us; skew 500 ppm
I20260806 03:42:20.468645 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:42:20.469434 712 heartbeater.cc:347] Connected to a master server at 127.30.138.126:46149
I20260806 03:42:20.469833 712 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:20.470762 712 heartbeater.cc:511] Master 127.30.138.126:46149 requested a full tablet report, sending...
I20260806 03:42:20.471876 31273 webserver.cc:460] Webserver started at http://127.30.138.67:39229/ using document root <none> and password file <none>
I20260806 03:42:20.472553 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:20.472798 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:20.473129 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:20.473061 403 ts_manager.cc:194] Registered new tserver with Master: d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:20.474865 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/instance:
uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.475409 403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35374
I20260806 03:42:20.480051 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 03:42:20.483575 726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.484357 31273 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 03:42:20.484654 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root
uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
format_stamp: "Formatted at 2026-08-06 03:42:20 on dist-test-slave-tk9z"
I20260806 03:42:20.484932 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:20.501284 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:20.502663 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:20.504169 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:20.506496 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:20.506683 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.506973 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:20.507107 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:20.550823 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.67:43937
I20260806 03:42:20.550940 811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.67:43937 every 8 connection(s)
I20260806 03:42:20.567276 816 heartbeater.cc:347] Connected to a master server at 127.30.138.126:46149
I20260806 03:42:20.567706 816 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:20.568397 816 heartbeater.cc:511] Master 127.30.138.126:46149 requested a full tablet report, sending...
I20260806 03:42:20.570464 403 ts_manager.cc:194] Registered new tserver with Master: 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:20.571219 31273 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017022907s
I20260806 03:42:20.572472 403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35388
I20260806 03:42:20.573753 31273 test_util.cc:268] Using random seed: -1395023867
I20260806 03:42:20.602535 403 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35392:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:42:20.608906 403 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:42:20.841305 541 tablet_service.cc:1467] Processing CreateTablet for tablet 651bac8237ca43fabc87a9897b0d7e51 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:42:20.842350 537 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:42:20.842895 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651bac8237ca43fabc87a9897b0d7e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.843751 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4c59f28ea45f6998c3f7f0b7dda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.845023 539 tablet_service.cc:1467] Processing CreateTablet for tablet 90b8bbd25b8646eb93f739aafb4d00f8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:42:20.845968 657 tablet_service.cc:1467] Processing CreateTablet for tablet 651bac8237ca43fabc87a9897b0d7e51 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:42:20.846419 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b8bbd25b8646eb93f739aafb4d00f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.847550 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651bac8237ca43fabc87a9897b0d7e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.848315 650 tablet_service.cc:1467] Processing CreateTablet for tablet e8b626bb795440e18f02bbdcbd10f9ce (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:42:20.849648 644 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbe5a0e51744568ce535d54093a890 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:42:20.850936 538 tablet_service.cc:1467] Processing CreateTablet for tablet dd5ae69655db414685b926cfecc43d65 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:42:20.852447 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5ae69655db414685b926cfecc43d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.853313 536 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba60616917a49ff9bdbfbe90b175c6c (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:42:20.855563 652 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:42:20.855573 534 tablet_service.cc:1467] Processing CreateTablet for tablet e8b626bb795440e18f02bbdcbd10f9ce (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:42:20.856913 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b626bb795440e18f02bbdcbd10f9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.856717 533 tablet_service.cc:1467] Processing CreateTablet for tablet 0488766430fc45ddb1edb22dd107266d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:42:20.858150 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488766430fc45ddb1edb22dd107266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.857959 651 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba60616917a49ff9bdbfbe90b175c6c (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:42:20.859632 532 tablet_service.cc:1467] Processing CreateTablet for tablet 2bea9147cd344c6d8e771c5d9968f2e7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:42:20.860355 649 tablet_service.cc:1467] Processing CreateTablet for tablet 0488766430fc45ddb1edb22dd107266d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:42:20.861616 646 tablet_service.cc:1467] Processing CreateTablet for tablet ba8119c361164a62a5ab1232752156bb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:42:20.861991 531 tablet_service.cc:1467] Processing CreateTablet for tablet ba8119c361164a62a5ab1232752156bb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:42:20.864099 648 tablet_service.cc:1467] Processing CreateTablet for tablet 2bea9147cd344c6d8e771c5d9968f2e7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:42:20.864343 530 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbe5a0e51744568ce535d54093a890 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:42:20.849679 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b626bb795440e18f02bbdcbd10f9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.866194 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488766430fc45ddb1edb22dd107266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.866919 652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4c59f28ea45f6998c3f7f0b7dda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.867571 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba60616917a49ff9bdbfbe90b175c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.867686 653 tablet_service.cc:1467] Processing CreateTablet for tablet dd5ae69655db414685b926cfecc43d65 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:42:20.869020 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5ae69655db414685b926cfecc43d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.870839 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bea9147cd344c6d8e771c5d9968f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.874715 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8119c361164a62a5ab1232752156bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.874835 656 tablet_service.cc:1467] Processing CreateTablet for tablet 90b8bbd25b8646eb93f739aafb4d00f8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:42:20.876199 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b8bbd25b8646eb93f739aafb4d00f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.878958 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbe5a0e51744568ce535d54093a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.882742 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8119c361164a62a5ab1232752156bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.886921 532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bea9147cd344c6d8e771c5d9968f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.887698 765 tablet_service.cc:1467] Processing CreateTablet for tablet 651bac8237ca43fabc87a9897b0d7e51 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:42:20.889087 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 651bac8237ca43fabc87a9897b0d7e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.890589 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba60616917a49ff9bdbfbe90b175c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.892987 764 tablet_service.cc:1467] Processing CreateTablet for tablet 90b8bbd25b8646eb93f739aafb4d00f8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:42:20.894774 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbe5a0e51744568ce535d54093a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.894979 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b8bbd25b8646eb93f739aafb4d00f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.901173 763 tablet_service.cc:1467] Processing CreateTablet for tablet dd5ae69655db414685b926cfecc43d65 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:42:20.903348 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5ae69655db414685b926cfecc43d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.908469 762 tablet_service.cc:1467] Processing CreateTablet for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:42:20.909999 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4c59f28ea45f6998c3f7f0b7dda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.915284 761 tablet_service.cc:1467] Processing CreateTablet for tablet 8ba60616917a49ff9bdbfbe90b175c6c (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:42:20.916828 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ba60616917a49ff9bdbfbe90b175c6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.941018 843 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:20.948865 843 tablet_bootstrap.cc:654] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:20.950711 760 tablet_service.cc:1467] Processing CreateTablet for tablet e8b626bb795440e18f02bbdcbd10f9ce (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:42:20.952592 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8b626bb795440e18f02bbdcbd10f9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.963238 762 tablet_service.cc:1467] Processing CreateTablet for tablet 0488766430fc45ddb1edb22dd107266d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:42:20.963608 759 tablet_service.cc:1467] Processing CreateTablet for tablet 2bea9147cd344c6d8e771c5d9968f2e7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:42:20.964638 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0488766430fc45ddb1edb22dd107266d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.965240 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bea9147cd344c6d8e771c5d9968f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:20.979908 843 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:20.980525 843 ts_tablet_manager.cc:1397] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.003s
I20260806 03:42:20.987735 651 tablet_service.cc:1467] Processing CreateTablet for tablet 329fbbb627464f478b8d2fd7a28851cc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:42:20.989482 656 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb062ea0544f0ab7202b067aaa662e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:42:20.991571 646 tablet_service.cc:1467] Processing CreateTablet for tablet d1e65c896c1a47c4a0a7eb7b1036bbba (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:42:20.989570 843 raft_consensus.cc:348] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:20.993213 843 raft_consensus.cc:374] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:20.993992 843 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:20.995527 843 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.007871 842 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:21.019132 845 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:21.019613 765 tablet_service.cc:1467] Processing CreateTablet for tablet ba8119c361164a62a5ab1232752156bb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:42:21.020975 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8119c361164a62a5ab1232752156bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.026546 534 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb062ea0544f0ab7202b067aaa662e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:42:21.028203 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb062ea0544f0ab7202b067aaa662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.026541 539 tablet_service.cc:1467] Processing CreateTablet for tablet 329fbbb627464f478b8d2fd7a28851cc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:42:21.031769 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 329fbbb627464f478b8d2fd7a28851cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.035436 531 tablet_service.cc:1467] Processing CreateTablet for tablet 921ac79ba4d94c039b073809e4a17f11 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:42:21.036455 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e65c896c1a47c4a0a7eb7b1036bbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.038678 653 tablet_service.cc:1467] Processing CreateTablet for tablet ce36e9a0e95c408ca319bfb41b336e31 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:42:21.039978 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce36e9a0e95c408ca319bfb41b336e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.042284 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921ac79ba4d94c039b073809e4a17f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.043895 652 tablet_service.cc:1467] Processing CreateTablet for tablet 93b5917270cd4146bb9b6c0ac0dc8414 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:42:21.045192 652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b5917270cd4146bb9b6c0ac0dc8414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.063016 816 heartbeater.cc:503] Master 127.30.138.126:46149 was elected leader, sending a full tablet report...
I20260806 03:42:21.062901 648 tablet_service.cc:1467] Processing CreateTablet for tablet b635b355749342d0ad7857884a590b8b (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:42:21.070748 843 ts_tablet_manager.cc:1428] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.090s user 0.032s sys 0.030s
I20260806 03:42:21.071918 843 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:21.076537 537 tablet_service.cc:1467] Processing CreateTablet for tablet 93b5917270cd4146bb9b6c0ac0dc8414 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:42:21.078017 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b5917270cd4146bb9b6c0ac0dc8414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.078491 843 tablet_bootstrap.cc:654] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.077175 536 tablet_service.cc:1467] Processing CreateTablet for tablet ce36e9a0e95c408ca319bfb41b336e31 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:42:21.079653 541 tablet_service.cc:1467] Processing CreateTablet for tablet b635b355749342d0ad7857884a590b8b (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:42:21.081251 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce36e9a0e95c408ca319bfb41b336e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.082551 763 tablet_service.cc:1467] Processing CreateTablet for tablet e1fbe5a0e51744568ce535d54093a890 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:42:21.082943 761 tablet_service.cc:1467] Processing CreateTablet for tablet 329fbbb627464f478b8d2fd7a28851cc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:42:21.083913 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbe5a0e51744568ce535d54093a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.084578 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 329fbbb627464f478b8d2fd7a28851cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.083966 758 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb062ea0544f0ab7202b067aaa662e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:42:21.086331 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb062ea0544f0ab7202b067aaa662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.090066 846 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:21.090713 846 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.074221 845 tablet_bootstrap.cc:654] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.089071 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb062ea0544f0ab7202b067aaa662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.097987 649 tablet_service.cc:1467] Processing CreateTablet for tablet ac21875cbf2a4ad6b9eee7859bc361fa (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:42:21.106724 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 329fbbb627464f478b8d2fd7a28851cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.108074 657 tablet_service.cc:1467] Processing CreateTablet for tablet f37d457494b64bb48d2f06013a9d26c0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 03:42:21.110672 846 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:21.112371 532 tablet_service.cc:1467] Processing CreateTablet for tablet ac21875cbf2a4ad6b9eee7859bc361fa (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:42:21.112313 756 tablet_service.cc:1467] Processing CreateTablet for tablet d1e65c896c1a47c4a0a7eb7b1036bbba (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:42:21.115581 755 tablet_service.cc:1467] Processing CreateTablet for tablet 921ac79ba4d94c039b073809e4a17f11 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:42:21.138118 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921ac79ba4d94c039b073809e4a17f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.141810 646 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc28108ed7945c88b5b7e3560f525e5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:42:21.143267 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc28108ed7945c88b5b7e3560f525e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.143462 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e65c896c1a47c4a0a7eb7b1036bbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.146811 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac21875cbf2a4ad6b9eee7859bc361fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.122628 644 tablet_service.cc:1467] Processing CreateTablet for tablet 921ac79ba4d94c039b073809e4a17f11 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:42:21.150143 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921ac79ba4d94c039b073809e4a17f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.152191 650 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0f8d511d8e4dce9973f791ed3bb702 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:42:21.152506 533 tablet_service.cc:1467] Processing CreateTablet for tablet d1e65c896c1a47c4a0a7eb7b1036bbba (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:42:21.155471 530 tablet_service.cc:1467] Processing CreateTablet for tablet f37d457494b64bb48d2f06013a9d26c0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
W20260806 03:42:21.156059 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:21.157312 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0f8d511d8e4dce9973f791ed3bb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.177073 842 tablet_bootstrap.cc:654] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.176988 537 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0f8d511d8e4dce9973f791ed3bb702 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:42:21.182548 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635b355749342d0ad7857884a590b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.185459 538 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc28108ed7945c88b5b7e3560f525e5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:42:21.189533 646 tablet_service.cc:1467] Processing CreateTablet for tablet 14cd85313f7a4ff7a17b8200a6317da4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:42:21.191005 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cd85313f7a4ff7a17b8200a6317da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.212960 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635b355749342d0ad7857884a590b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.216933 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37d457494b64bb48d2f06013a9d26c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.216660 765 tablet_service.cc:1467] Processing CreateTablet for tablet b635b355749342d0ad7857884a590b8b (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:42:21.217932 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635b355749342d0ad7857884a590b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.218500 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e65c896c1a47c4a0a7eb7b1036bbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.219687 539 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9afce5efa4491797ebed2aab67532f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:42:21.219956 759 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc28108ed7945c88b5b7e3560f525e5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:42:21.221199 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc28108ed7945c88b5b7e3560f525e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.222131 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37d457494b64bb48d2f06013a9d26c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.166267 652 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9afce5efa4491797ebed2aab67532f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:42:21.227448 652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9afce5efa4491797ebed2aab67532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.229437 656 tablet_service.cc:1467] Processing CreateTablet for tablet fc95e3152e264afab52ba5a7cd5458a7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:42:21.229897 845 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:21.231715 845 ts_tablet_manager.cc:1397] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.213s user 0.031s sys 0.006s
I20260806 03:42:21.238744 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0f8d511d8e4dce9973f791ed3bb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.231976 760 tablet_service.cc:1467] Processing CreateTablet for tablet ce36e9a0e95c408ca319bfb41b336e31 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:42:21.239984 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce36e9a0e95c408ca319bfb41b336e31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.241438 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc28108ed7945c88b5b7e3560f525e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.244150 845 raft_consensus.cc:348] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.247356 845 raft_consensus.cc:374] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.247716 845 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.250128 532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac21875cbf2a4ad6b9eee7859bc361fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.263574 653 tablet_service.cc:1467] Processing CreateTablet for tablet c9369dafb3f342e7a15d44086a377b55 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:42:21.264935 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9369dafb3f342e7a15d44086a377b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.266016 534 tablet_service.cc:1467] Processing CreateTablet for tablet 14cd85313f7a4ff7a17b8200a6317da4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:42:21.267410 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cd85313f7a4ff7a17b8200a6317da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.267994 842 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:21.268466 842 ts_tablet_manager.cc:1397] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.261s user 0.008s sys 0.009s
I20260806 03:42:21.272662 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9afce5efa4491797ebed2aab67532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.271733 651 tablet_service.cc:1467] Processing CreateTablet for tablet ae397e5cb5a14531a20bf02d8c71254d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:42:21.276284 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae397e5cb5a14531a20bf02d8c71254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.278774 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc95e3152e264afab52ba5a7cd5458a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.287492 842 raft_consensus.cc:348] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.288298 842 raft_consensus.cc:374] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.288226 763 tablet_service.cc:1467] Processing CreateTablet for tablet ac21875cbf2a4ad6b9eee7859bc361fa (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:42:21.288681 842 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.289636 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac21875cbf2a4ad6b9eee7859bc361fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.289410 842 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.296962 843 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:21.297477 843 ts_tablet_manager.cc:1397] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.226s user 0.031s sys 0.052s
I20260806 03:42:21.300220 843 raft_consensus.cc:348] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.301002 843 raft_consensus.cc:374] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.270076 646 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b590a95fa411a91b9a3818bf7d29f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:42:21.302146 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b590a95fa411a91b9a3818bf7d29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.307402 657 tablet_service.cc:1467] Processing CreateTablet for tablet 721de14efa984f9ebf40c801a835c6ac (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:42:21.311231 758 tablet_service.cc:1467] Processing CreateTablet for tablet f37d457494b64bb48d2f06013a9d26c0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 03:42:21.316082 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37d457494b64bb48d2f06013a9d26c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.316888 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0f8d511d8e4dce9973f791ed3bb702 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:42:21.319067 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721de14efa984f9ebf40c801a835c6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.321111 843 raft_consensus.cc:729] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.322149 843 consensus_queue.cc:260] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.328811 537 tablet_service.cc:1467] Processing CreateTablet for tablet fc95e3152e264afab52ba5a7cd5458a7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:42:21.339143 764 tablet_service.cc:1467] Processing CreateTablet for tablet 93b5917270cd4146bb9b6c0ac0dc8414 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:42:21.330228 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc95e3152e264afab52ba5a7cd5458a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.344844 644 tablet_service.cc:1467] Processing CreateTablet for tablet 20fcbc41710a42cd9ee5dacc5d9577fb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:42:21.286665 845 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.318217 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0f8d511d8e4dce9973f791ed3bb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.367558 595 heartbeater.cc:503] Master 127.30.138.126:46149 was elected leader, sending a full tablet report...
I20260806 03:42:21.382959 843 ts_tablet_manager.cc:1428] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.085s user 0.004s sys 0.004s
I20260806 03:42:21.383584 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fcbc41710a42cd9ee5dacc5d9577fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.385257 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:21.391947 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:21.397356 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93b5917270cd4146bb9b6c0ac0dc8414. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:42:21.407110 727 leader_election.cc:341] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:21.408006 727 leader_election.cc:341] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:21.408420 727 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:21.409199 846 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:21.412667 539 tablet_service.cc:1467] Processing CreateTablet for tablet c9369dafb3f342e7a15d44086a377b55 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:42:21.414011 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9369dafb3f342e7a15d44086a377b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.415458 842 ts_tablet_manager.cc:1428] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.147s user 0.009s sys 0.014s
W20260806 03:42:21.365727 378 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 47135.1us
I20260806 03:42:21.417774 649 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcd22bae8db4028ba3d0e499cd18e13 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:42:21.419101 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcd22bae8db4028ba3d0e499cd18e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.432174 763 tablet_service.cc:1467] Processing CreateTablet for tablet 14cd85313f7a4ff7a17b8200a6317da4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:42:21.432300 539 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b590a95fa411a91b9a3818bf7d29f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:42:21.433585 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14cd85313f7a4ff7a17b8200a6317da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.433593 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b590a95fa411a91b9a3818bf7d29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.436591 759 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9afce5efa4491797ebed2aab67532f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:42:21.437912 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9afce5efa4491797ebed2aab67532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.440076 761 tablet_service.cc:1467] Processing CreateTablet for tablet fc95e3152e264afab52ba5a7cd5458a7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:42:21.441401 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc95e3152e264afab52ba5a7cd5458a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.403599 843 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:21.416489 842 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:21.446638 653 tablet_service.cc:1467] Processing CreateTablet for tablet 604c561db3f3435da75d2536381f6b96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:42:21.461026 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604c561db3f3435da75d2536381f6b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.451435 762 tablet_service.cc:1467] Processing CreateTablet for tablet c9369dafb3f342e7a15d44086a377b55 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:42:21.472076 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9369dafb3f342e7a15d44086a377b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.478353 712 heartbeater.cc:503] Master 127.30.138.126:46149 was elected leader, sending a full tablet report...
I20260806 03:42:21.479010 845 ts_tablet_manager.cc:1428] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.241s user 0.014s sys 0.000s
I20260806 03:42:21.495319 651 tablet_service.cc:1467] Processing CreateTablet for tablet 55d4d892dae44f9b919acae13e8d6642 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:42:21.496641 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d4d892dae44f9b919acae13e8d6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.500679 530 tablet_service.cc:1467] Processing CreateTablet for tablet 721de14efa984f9ebf40c801a835c6ac (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:42:21.508517 533 tablet_service.cc:1467] Processing CreateTablet for tablet 20fcbc41710a42cd9ee5dacc5d9577fb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:42:21.507408 756 tablet_service.cc:1467] Processing CreateTablet for tablet 721de14efa984f9ebf40c801a835c6ac (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:42:21.507346 755 tablet_service.cc:1467] Processing CreateTablet for tablet ce3b590a95fa411a91b9a3818bf7d29f (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:42:21.511650 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b590a95fa411a91b9a3818bf7d29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.513281 765 tablet_service.cc:1467] Processing CreateTablet for tablet 20fcbc41710a42cd9ee5dacc5d9577fb (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:42:21.541853 536 tablet_service.cc:1467] Processing CreateTablet for tablet 55d4d892dae44f9b919acae13e8d6642 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:42:21.514359 764 tablet_service.cc:1467] Processing CreateTablet for tablet ae397e5cb5a14531a20bf02d8c71254d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:42:21.502321 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721de14efa984f9ebf40c801a835c6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.548511 846 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:21.548995 541 tablet_service.cc:1467] Processing CreateTablet for tablet 864f4306a7964687b312ccf31fe6797a (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:42:21.549062 846 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.551237 846 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:21.553565 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:21.553498 758 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcd22bae8db4028ba3d0e499cd18e13 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:42:21.555027 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:21.556833 727 leader_election.cc:341] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:21.556756 538 tablet_service.cc:1467] Processing CreateTablet for tablet 760b94c8ff89432c8cbbb7ed61be23b1 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260806 03:42:21.557588 727 leader_election.cc:341] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:21.557996 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:21.559125 846 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:21.514926 845 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:21.515574 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 721de14efa984f9ebf40c801a835c6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.517856 657 tablet_service.cc:1467] Processing CreateTablet for tablet 864f4306a7964687b312ccf31fe6797a (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:42:21.560768 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864f4306a7964687b312ccf31fe6797a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.561244 650 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:42:21.561633 762 tablet_service.cc:1467] Processing CreateTablet for tablet 604c561db3f3435da75d2536381f6b96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:42:21.565352 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcb8ef1a79d4601b243ec9ce6e6acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.531534 649 tablet_service.cc:1467] Processing CreateTablet for tablet 760b94c8ff89432c8cbbb7ed61be23b1 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 03:42:21.535305 532 tablet_service.cc:1467] Processing CreateTablet for tablet ae397e5cb5a14531a20bf02d8c71254d (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:42:21.538118 843 tablet_bootstrap.cc:654] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.571242 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760b94c8ff89432c8cbbb7ed61be23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.530567 534 tablet_service.cc:1467] Processing CreateTablet for tablet 604c561db3f3435da75d2536381f6b96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:42:21.513374 531 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcd22bae8db4028ba3d0e499cd18e13 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:42:21.562866 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae397e5cb5a14531a20bf02d8c71254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.575639 842 tablet_bootstrap.cc:654] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.576557 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604c561db3f3435da75d2536381f6b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.578660 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fcbc41710a42cd9ee5dacc5d9577fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.581243 759 tablet_service.cc:1467] Processing CreateTablet for tablet 55d4d892dae44f9b919acae13e8d6642 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:42:21.582613 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d4d892dae44f9b919acae13e8d6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.583292 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcd22bae8db4028ba3d0e499cd18e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.586432 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864f4306a7964687b312ccf31fe6797a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.532788 651 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcd22a4b7694c71b676f8939a238cd0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:42:21.587539 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcd22a4b7694c71b676f8939a238cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.587672 648 tablet_service.cc:1467] Processing CreateTablet for tablet b19e6696d8d549f58265db62b449e706 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:42:21.589005 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19e6696d8d549f58265db62b449e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.590708 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcd22bae8db4028ba3d0e499cd18e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.592298 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fcbc41710a42cd9ee5dacc5d9577fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.598057 539 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:42:21.598906 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604c561db3f3435da75d2536381f6b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.599689 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcb8ef1a79d4601b243ec9ce6e6acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.608775 760 tablet_service.cc:1467] Processing CreateTablet for tablet 864f4306a7964687b312ccf31fe6797a (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:42:21.609021 761 tablet_service.cc:1467] Processing CreateTablet for tablet 760b94c8ff89432c8cbbb7ed61be23b1 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 03:42:21.609324 854 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:21.609870 854 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.610621 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864f4306a7964687b312ccf31fe6797a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.613951 854 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:21.585517 537 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcd22a4b7694c71b676f8939a238cd0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:42:21.617411 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760b94c8ff89432c8cbbb7ed61be23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.650631 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d4d892dae44f9b919acae13e8d6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.652096 532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae397e5cb5a14531a20bf02d8c71254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.652618 842 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:21.653446 842 ts_tablet_manager.cc:1397] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.237s user 0.009s sys 0.000s
I20260806 03:42:21.653599 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760b94c8ff89432c8cbbb7ed61be23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.655467 763 tablet_service.cc:1467] Processing CreateTablet for tablet 3bcd22a4b7694c71b676f8939a238cd0 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:42:21.656616 761 tablet_service.cc:1467] Processing CreateTablet for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:42:21.657876 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcd22a4b7694c71b676f8939a238cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.656483 842 raft_consensus.cc:348] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.658599 842 raft_consensus.cc:374] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.658802 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bcd22a4b7694c71b676f8939a238cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.659029 842 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.659999 842 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.663949 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcb8ef1a79d4601b243ec9ce6e6acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.676645 843 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:21.677205 843 ts_tablet_manager.cc:1397] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.286s user 0.005s sys 0.010s
I20260806 03:42:21.680094 843 raft_consensus.cc:348] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:21.680886 843 raft_consensus.cc:374] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.681180 843 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.681978 843 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:21.691026 644 tablet_service.cc:1467] Processing CreateTablet for tablet 44d06aad0812478bb2270a2c80917306 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:42:21.691854 843 ts_tablet_manager.cc:1428] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260806 03:42:21.692647 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d06aad0812478bb2270a2c80917306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.693190 843 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:21.708158 842 ts_tablet_manager.cc:1428] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.054s user 0.008s sys 0.002s
I20260806 03:42:21.708488 530 tablet_service.cc:1467] Processing CreateTablet for tablet b19e6696d8d549f58265db62b449e706 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:42:21.709210 842 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:21.715564 657 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ac3ebccff4054953475f839218e45 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:42:21.716995 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ac3ebccff4054953475f839218e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.719208 650 tablet_service.cc:1467] Processing CreateTablet for tablet 25056084174744caba1a79c3eb002598 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:42:21.720497 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25056084174744caba1a79c3eb002598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.716619 755 tablet_service.cc:1467] Processing CreateTablet for tablet 44d06aad0812478bb2270a2c80917306 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:42:21.735363 855 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:21.735899 855 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.740444 855 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:21.747134 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d06aad0812478bb2270a2c80917306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.759791 845 tablet_bootstrap.cc:654] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.748028 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19e6696d8d549f58265db62b449e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.775713 764 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ac3ebccff4054953475f839218e45 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:42:21.797857 759 tablet_service.cc:1467] Processing CreateTablet for tablet 25056084174744caba1a79c3eb002598 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:42:21.806991 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:21.810078 498 leader_election.cc:341] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:21.791693 758 tablet_service.cc:1467] Processing CreateTablet for tablet b19e6696d8d549f58265db62b449e706 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:42:21.789865 843 tablet_bootstrap.cc:654] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.849258 762 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec448ca686a4641a5af84944f63b7e2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 03:42:21.852553 765 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9e5190380411684128482e3a899b8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:42:21.831424 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ac3ebccff4054953475f839218e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.864971 531 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8ac3ebccff4054953475f839218e45 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:42:21.866063 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9e5190380411684128482e3a899b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.878314 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19e6696d8d549f58265db62b449e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.879535 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25056084174744caba1a79c3eb002598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.880723 845 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:21.881253 845 ts_tablet_manager.cc:1397] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.367s user 0.022s sys 0.023s
I20260806 03:42:21.866338 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ac3ebccff4054953475f839218e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.884066 845 raft_consensus.cc:348] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.884898 845 raft_consensus.cc:374] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:21.885280 845 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:21.886147 845 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:21.888967 845 ts_tablet_manager.cc:1428] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:42:21.872465 760 tablet_service.cc:1467] Processing CreateTablet for tablet 30edb33481c941c08c9a8430b4d0bc54 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:42:21.897456 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edb33481c941c08c9a8430b4d0bc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.907603 539 tablet_service.cc:1467] Processing CreateTablet for tablet 25056084174744caba1a79c3eb002598 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:42:21.909006 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25056084174744caba1a79c3eb002598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.914718 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec448ca686a4641a5af84944f63b7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.915257 653 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec448ca686a4641a5af84944f63b7e2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 03:42:21.916575 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec448ca686a4641a5af84944f63b7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.917903 531 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9e5190380411684128482e3a899b8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:42:21.919267 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9e5190380411684128482e3a899b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.924916 842 tablet_bootstrap.cc:654] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:21.927951 534 tablet_service.cc:1467] Processing CreateTablet for tablet 44d06aad0812478bb2270a2c80917306 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:42:21.929318 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d06aad0812478bb2270a2c80917306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.929944 648 tablet_service.cc:1467] Processing CreateTablet for tablet 30edb33481c941c08c9a8430b4d0bc54 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:42:21.931255 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edb33481c941c08c9a8430b4d0bc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.935078 756 tablet_service.cc:1467] Processing CreateTablet for tablet aa826ec82b0a402580f530e7eb832852 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:42:21.935770 531 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 03:42:21.936462 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa826ec82b0a402580f530e7eb832852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.937016 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbf67cdfa6f4f2785238e161fd2c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.953315 763 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 03:42:21.953027 536 tablet_service.cc:1467] Processing CreateTablet for tablet 7ec448ca686a4641a5af84944f63b7e2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 03:42:21.945647 538 tablet_service.cc:1467] Processing CreateTablet for tablet 30edb33481c941c08c9a8430b4d0bc54 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:42:21.962178 656 tablet_service.cc:1467] Processing CreateTablet for tablet 43a6bdb2bc794303966a31eeb84328e9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:42:21.972878 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a6bdb2bc794303966a31eeb84328e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.977114 541 tablet_service.cc:1467] Processing CreateTablet for tablet aa826ec82b0a402580f530e7eb832852 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:42:21.962630 646 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2a8e791d4943d9a81d8d8c75d09783 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:42:21.963303 649 tablet_service.cc:1467] Processing CreateTablet for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 03:42:21.964108 652 tablet_service.cc:1467] Processing CreateTablet for tablet 20df53c445204494999245a03bc56a91 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 03:42:21.972878 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30edb33481c941c08c9a8430b4d0bc54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.966980 761 tablet_service.cc:1467] Processing CreateTablet for tablet 43a6bdb2bc794303966a31eeb84328e9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:42:21.968928 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:21.982553 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ec448ca686a4641a5af84944f63b7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.983515 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a6bdb2bc794303966a31eeb84328e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.986094 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa826ec82b0a402580f530e7eb832852. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:42:21.986838 614 leader_election.cc:341] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:21.971457 644 tablet_service.cc:1467] Processing CreateTablet for tablet aa826ec82b0a402580f530e7eb832852 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:42:21.980145 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2a8e791d4943d9a81d8d8c75d09783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.993086 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbf67cdfa6f4f2785238e161fd2c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.993778 652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20df53c445204494999245a03bc56a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.995024 657 tablet_service.cc:1467] Processing CreateTablet for tablet 2018f00b21544838b05e402c218b85ea (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 03:42:21.996291 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2018f00b21544838b05e402c218b85ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:21.998983 531 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2a8e791d4943d9a81d8d8c75d09783 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:42:22.000317 842 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:21.959259 845 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.003931 842 ts_tablet_manager.cc:1397] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.295s user 0.009s sys 0.006s
I20260806 03:42:22.006306 843 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.006894 843 ts_tablet_manager.cc:1397] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.314s user 0.022s sys 0.001s
I20260806 03:42:22.012291 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.012840 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.012928 842 raft_consensus.cc:348] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.013782 842 raft_consensus.cc:374] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.014109 842 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
W20260806 03:42:22.014161 501 leader_election.cc:341] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.014688 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:42:22.014683 616 leader_election.cc:341] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.016639 616 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:22.016819 854 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.017409 855 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.022826 763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbf67cdfa6f4f2785238e161fd2c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.016172 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa826ec82b0a402580f530e7eb832852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.035427 532 tablet_service.cc:1467] Processing CreateTablet for tablet 43a6bdb2bc794303966a31eeb84328e9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:42:22.056798 846 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.057773 846 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.116384 758 tablet_service.cc:1467] Processing CreateTablet for tablet 20df53c445204494999245a03bc56a91 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 03:42:22.116842 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.117326 846 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.045109 651 tablet_service.cc:1467] Processing CreateTablet for tablet 68a9e5190380411684128482e3a899b8 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:42:22.117774 758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20df53c445204494999245a03bc56a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.118160 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.118717 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9e5190380411684128482e3a899b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:42:22.120102 727 leader_election.cc:341] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:22.122722 727 leader_election.cc:341] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.123188 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.126693 842 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.127305 854 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.127817 854 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.130000 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2018f00b21544838b05e402c218b85ea (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 03:42:22.104648 845 tablet_bootstrap.cc:654] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.108510 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9d2a8e791d4943d9a81d8d8c75d09783 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:42:22.110297 531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2a8e791d4943d9a81d8d8c75d09783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.009747 843 raft_consensus.cc:348] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.131229 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.131834 656 tablet_service.cc:1467] Processing CreateTablet for tablet 736f0260dd3a4e77a6115e55fa0c4269 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 03:42:22.162088 656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f0260dd3a4e77a6115e55fa0c4269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.133453 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.130043 854 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.164759 534 tablet_service.cc:1467] Processing CreateTablet for tablet 736f0260dd3a4e77a6115e55fa0c4269 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260806 03:42:22.166816 498 leader_election.cc:341] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.167164 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2018f00b21544838b05e402c218b85ea. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:42:22.167747 501 leader_election.cc:341] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.168200 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.139447 648 tablet_service.cc:1467] Processing CreateTablet for tablet fb67a142feaf496f862fbd7f06a925f9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 03:42:22.139911 538 tablet_service.cc:1467] Processing CreateTablet for tablet 20df53c445204494999245a03bc56a91 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 03:42:22.140300 846 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.140192 537 tablet_service.cc:1467] Processing CreateTablet for tablet 2018f00b21544838b05e402c218b85ea (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 03:42:22.080966 650 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5936af1dcf436eb5f06e8fc0765e83 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:42:22.169505 854 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.170784 648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb67a142feaf496f862fbd7f06a925f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.144141 530 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5936af1dcf436eb5f06e8fc0765e83 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:42:22.171156 843 raft_consensus.cc:374] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.171571 843 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.172590 843 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.174120 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2a8e791d4943d9a81d8d8c75d09783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.145512 653 tablet_service.cc:1467] Processing CreateTablet for tablet e12cd5292fc847be9a0e9d515b7faf96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:42:22.186363 650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5936af1dcf436eb5f06e8fc0765e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.147159 759 tablet_service.cc:1467] Processing CreateTablet for tablet fb67a142feaf496f862fbd7f06a925f9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 03:42:22.150795 541 tablet_service.cc:1467] Processing CreateTablet for tablet e12cd5292fc847be9a0e9d515b7faf96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:42:22.155916 533 tablet_service.cc:1467] Processing CreateTablet for tablet fb67a142feaf496f862fbd7f06a925f9 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260806 03:42:22.160422 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:22.168013 657 tablet_service.cc:1467] Processing CreateTablet for tablet 497d006930504e8184f1e5e512ade026 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 03:42:22.142426 762 tablet_service.cc:1467] Processing CreateTablet for tablet e12cd5292fc847be9a0e9d515b7faf96 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:42:22.134552 765 tablet_service.cc:1467] Processing CreateTablet for tablet 2c5936af1dcf436eb5f06e8fc0765e83 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:42:22.175438 756 tablet_service.cc:1467] Processing CreateTablet for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 03:42:22.193812 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497d006930504e8184f1e5e512ade026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.199967 532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43a6bdb2bc794303966a31eeb84328e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.201735 764 tablet_service.cc:1467] Processing CreateTablet for tablet 736f0260dd3a4e77a6115e55fa0c4269 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 03:42:22.205837 855 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.206339 855 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.210261 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12cd5292fc847be9a0e9d515b7faf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.190481 759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb67a142feaf496f862fbd7f06a925f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.214973 762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12cd5292fc847be9a0e9d515b7faf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.215615 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a414fbc8cc4236a2f89a5bec69c6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.215838 855 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.217610 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.218269 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5936af1dcf436eb5f06e8fc0765e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.219096 539 tablet_service.cc:1467] Processing CreateTablet for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20260806 03:42:22.220624 616 leader_election.cc:341] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.219802 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:22.225786 614 leader_election.cc:341] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.226213 614 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:22.227563 855 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.222627 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f0260dd3a4e77a6115e55fa0c4269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.234645 842 ts_tablet_manager.cc:1428] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.230s user 0.010s sys 0.001s
I20260806 03:42:22.235816 842 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.223073 843 ts_tablet_manager.cc:1428] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.216s user 0.008s sys 0.001s
I20260806 03:42:22.243381 843 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.240293 646 tablet_service.cc:1467] Processing CreateTablet for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 03:42:22.250622 845 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.251291 845 ts_tablet_manager.cc:1397] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.307s user 0.009s sys 0.016s
I20260806 03:42:22.251564 646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a414fbc8cc4236a2f89a5bec69c6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.250578 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f0260dd3a4e77a6115e55fa0c4269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.254097 845 raft_consensus.cc:348] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.254968 845 raft_consensus.cc:374] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.255280 845 raft_consensus.cc:729] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.256021 538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20df53c445204494999245a03bc56a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.256111 845 consensus_queue.cc:260] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.257452 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2018f00b21544838b05e402c218b85ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.260984 530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c5936af1dcf436eb5f06e8fc0765e83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.261651 533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb67a142feaf496f862fbd7f06a925f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.264524 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a414fbc8cc4236a2f89a5bec69c6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.267239 649 tablet_service.cc:1467] Processing CreateTablet for tablet 5502493e599c44abbc15f805d5298dbc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:42:22.268409 541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e12cd5292fc847be9a0e9d515b7faf96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.268757 649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502493e599c44abbc15f805d5298dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.269626 846 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.270114 846 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.272680 846 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.274981 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.275151 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:22.276397 727 leader_election.cc:341] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:22.277092 727 leader_election.cc:341] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.277503 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.278609 845 ts_tablet_manager.cc:1428] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20260806 03:42:22.278785 846 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.281685 536 tablet_service.cc:1467] Processing CreateTablet for tablet 497d006930504e8184f1e5e512ade026 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 03:42:22.294955 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497d006930504e8184f1e5e512ade026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.303580 644 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b55970984ed5ad958ef3c517c6b5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:42:22.285604 843 tablet_bootstrap.cc:654] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.288142 761 tablet_service.cc:1467] Processing CreateTablet for tablet 5502493e599c44abbc15f805d5298dbc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:42:22.291407 652 tablet_service.cc:1467] Processing CreateTablet for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:42:22.312937 644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b55970984ed5ad958ef3c517c6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.325034 845 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.328001 534 tablet_service.cc:1467] Processing CreateTablet for tablet 5502493e599c44abbc15f805d5298dbc (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:42:22.329385 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502493e599c44abbc15f805d5298dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.336745 842 tablet_bootstrap.cc:654] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.339341 652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.348111 657 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7b59fcc4bf4cafad47c9aed88c416e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:42:22.349529 657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7b59fcc4bf4cafad47c9aed88c416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.353816 539 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b55970984ed5ad958ef3c517c6b5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:42:22.348111 534 tablet_service.cc:1467] Processing CreateTablet for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:42:22.355334 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b55970984ed5ad958ef3c517c6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.361006 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502493e599c44abbc15f805d5298dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.361486 534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.364679 536 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7b59fcc4bf4cafad47c9aed88c416e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:42:22.365391 764 tablet_service.cc:1467] Processing CreateTablet for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:42:22.365962 536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7b59fcc4bf4cafad47c9aed88c416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.367136 755 tablet_service.cc:1467] Processing CreateTablet for tablet 497d006930504e8184f1e5e512ade026 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 03:42:22.367512 765 tablet_service.cc:1467] Processing CreateTablet for tablet ecf1b55970984ed5ad958ef3c517c6b5 (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:42:22.368270 764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.377324 755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497d006930504e8184f1e5e512ade026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.389194 765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b55970984ed5ad958ef3c517c6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.399482 845 tablet_bootstrap.cc:654] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.400478 760 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7b59fcc4bf4cafad47c9aed88c416e (DEFAULT_TABLE table=test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:42:22.401777 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7b59fcc4bf4cafad47c9aed88c416e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:22.420316 842 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.420874 842 ts_tablet_manager.cc:1397] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.011s
I20260806 03:42:22.423676 842 raft_consensus.cc:348] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.424429 842 raft_consensus.cc:374] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.424721 842 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.434945 842 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.441181 842 ts_tablet_manager.cc:1428] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260806 03:42:22.442461 842 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.443902 879 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.444428 879 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.448763 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:22.452939 501 leader_election.cc:341] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.453450 845 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.454063 845 ts_tablet_manager.cc:1397] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.129s user 0.013s sys 0.000s
I20260806 03:42:22.450093 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.454695 674 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:22.456195 498 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:22.457227 854 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.457600 854 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.457932 854 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.451645 879 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.465984 843 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.466589 843 ts_tablet_manager.cc:1397] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.223s user 0.021s sys 0.034s
I20260806 03:42:22.469712 843 raft_consensus.cc:348] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.470602 843 raft_consensus.cc:374] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.470924 843 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.471709 843 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.474179 843 ts_tablet_manager.cc:1428] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 03:42:22.476009 843 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.475708 845 raft_consensus.cc:348] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.476822 845 raft_consensus.cc:374] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.476872 855 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.477205 845 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.477370 855 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.478056 845 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.474207 842 tablet_bootstrap.cc:654] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.479848 855 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.482771 843 tablet_bootstrap.cc:654] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.483264 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.484133 775 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:22.485721 616 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:22.494336 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.494831 842 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.495286 842 ts_tablet_manager.cc:1397] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.008s
I20260806 03:42:22.482120 854 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:22.496824 614 leader_election.cc:341] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.502014 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.502288 842 raft_consensus.cc:348] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.502737 674 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.503774 855 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.510535 855 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.510915 674 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:22.511065 855 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.512250 498 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:22.513293 879 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.514106 842 raft_consensus.cc:374] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.514472 842 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.516289 842 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.519079 843 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.519760 843 ts_tablet_manager.cc:1397] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.024s
I20260806 03:42:22.526533 842 ts_tablet_manager.cc:1428] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.031s user 0.008s sys 0.002s
I20260806 03:42:22.528766 843 raft_consensus.cc:348] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.529994 843 raft_consensus.cc:374] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.530439 842 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.531746 879 raft_consensus.cc:686] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:22.532869 854 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.532575 879 consensus_queue.cc:237] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.534193 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:22.530483 843 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.535563 843 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:22.536147 501 leader_election.cc:341] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.537904 843 ts_tablet_manager.cc:1428] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260806 03:42:22.541762 855 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.542996 845 ts_tablet_manager.cc:1428] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.089s user 0.013s sys 0.012s
I20260806 03:42:22.544086 845 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.544785 843 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.547399 402 catalog_manager.cc:5495] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:22.549010 846 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.549526 846 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.551867 846 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.554077 842 tablet_bootstrap.cc:654] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.554338 843 tablet_bootstrap.cc:654] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.557857 845 tablet_bootstrap.cc:654] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.560523 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.562202 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:22.565204 727 leader_election.cc:341] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:22.566056 727 leader_election.cc:341] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.566591 727 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.572729 843 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.573540 843 ts_tablet_manager.cc:1397] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.009s
I20260806 03:42:22.577382 843 raft_consensus.cc:348] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.578588 843 raft_consensus.cc:374] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.579043 843 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.578616 846 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:22.580117 843 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.583138 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:22.583611 843 ts_tablet_manager.cc:1428] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
W20260806 03:42:22.584807 614 leader_election.cc:341] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.587628 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:22.588337 775 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.588755 843 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.594641 846 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.595300 846 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.597786 843 tablet_bootstrap.cc:654] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.599274 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.600196 564 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.601734 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.623752 846 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.624838 846 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.625276 846 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.625674 846 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.630949 855 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.639312 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:22.641284 727 leader_election.cc:341] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.642247 775 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:22.643661 616 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:22.645116 855 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.646979 855 raft_consensus.cc:686] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:22.648169 855 consensus_queue.cc:237] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.653698 901 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.654387 901 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.658075 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.658962 564 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.660645 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.665576 842 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.666100 842 ts_tablet_manager.cc:1397] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.136s user 0.030s sys 0.038s
I20260806 03:42:22.668785 846 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.668933 842 raft_consensus.cc:348] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.669692 842 raft_consensus.cc:374] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.669998 842 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.671319 901 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.672395 845 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.672960 845 ts_tablet_manager.cc:1397] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.129s user 0.030s sys 0.026s
I20260806 03:42:22.675633 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.675966 845 raft_consensus.cc:348] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.676746 845 raft_consensus.cc:374] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.677109 845 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.672335 901 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.677783 901 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.678138 901 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.677977 845 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.679121 674 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.679404 842 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.682919 842 ts_tablet_manager.cc:1428] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260806 03:42:22.684280 842 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.684460 845 ts_tablet_manager.cc:1428] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260806 03:42:22.685462 845 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.689493 843 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.690105 843 ts_tablet_manager.cc:1397] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.102s user 0.049s sys 0.044s
I20260806 03:42:22.691650 845 tablet_bootstrap.cc:654] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.691709 842 tablet_bootstrap.cc:654] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.693564 843 raft_consensus.cc:348] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.694413 843 raft_consensus.cc:374] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.694747 843 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.696101 846 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.695704 843 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.696849 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:22.697511 564 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.698897 843 ts_tablet_manager.cc:1428] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260806 03:42:22.703320 843 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.704067 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:22.705849 727 leader_election.cc:341] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:22.705741 713 tablet.cc:2369] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:42:22.708823 403 catalog_manager.cc:5495] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:22.709939 564 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:22.715394 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.716185 843 tablet_bootstrap.cc:654] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.720665 845 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.721171 845 ts_tablet_manager.cc:1397] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.007s
I20260806 03:42:22.723824 846 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.724903 846 raft_consensus.cc:686] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:22.725914 846 consensus_queue.cc:237] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.729097 845 raft_consensus.cc:348] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.729933 845 raft_consensus.cc:374] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.730262 845 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.731082 845 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.729493 901 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.733651 845 ts_tablet_manager.cc:1428] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 03:42:22.736737 843 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.737098 901 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.737214 843 ts_tablet_manager.cc:1397] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.008s
I20260806 03:42:22.738010 842 tablet_bootstrap.cc:492] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.738595 842 ts_tablet_manager.cc:1397] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.055s user 0.028s sys 0.025s
I20260806 03:42:22.738718 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:22.739432 564 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.740226 843 raft_consensus.cc:348] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.740955 843 raft_consensus.cc:374] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.741225 843 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.741940 842 raft_consensus.cc:348] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.741999 843 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.742928 842 raft_consensus.cc:374] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.743374 842 raft_consensus.cc:729] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.743322 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.744069 674 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.744644 843 ts_tablet_manager.cc:1428] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:42:22.744431 842 consensus_queue.cc:260] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.745692 843 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.747298 842 ts_tablet_manager.cc:1428] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260806 03:42:22.748472 842 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.752029 843 tablet_bootstrap.cc:654] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.736620 845 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.755690 842 tablet_bootstrap.cc:654] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.759248 564 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:22.761490 879 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.762104 879 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.766115 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.767012 775 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:22.768522 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.769443 854 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.770597 854 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.771032 854 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.769780 879 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.777697 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:22.779904 498 leader_election.cc:341] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.781944 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.784825 842 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.785468 842 ts_tablet_manager.cc:1397] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.015s
I20260806 03:42:22.789237 842 raft_consensus.cc:348] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.789976 845 tablet_bootstrap.cc:654] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.790196 842 raft_consensus.cc:374] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.790618 842 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.790714 846 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.791282 846 raft_consensus.cc:686] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:22.790743 854 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.791522 842 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.792194 846 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.794662 842 ts_tablet_manager.cc:1428] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 03:42:22.795836 842 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.795673 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:22.797086 498 leader_election.cc:341] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.776494 674 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:22.803205 842 tablet_bootstrap.cc:654] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.779835 400 catalog_manager.cc:5495] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:22.807443 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:22.808076 775 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.802675 854 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.827731 854 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.828287 854 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.830281 843 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.831009 843 ts_tablet_manager.cc:1397] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.086s user 0.028s sys 0.042s
I20260806 03:42:22.831233 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.831943 776 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:22.833233 501 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.827879 775 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:22.834102 879 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.834538 879 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.834903 879 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.835033 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.835636 843 raft_consensus.cc:348] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.836339 889 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.836691 843 raft_consensus.cc:374] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.836875 889 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:22.837112 843 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.837839 889 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.838232 843 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.841142 843 ts_tablet_manager.cc:1428] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260806 03:42:22.842676 843 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.843314 854 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.844713 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:22.846131 498 leader_election.cc:341] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.847231 854 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.847694 854 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.850471 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.851013 843 tablet_bootstrap.cc:654] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:22.851958 498 leader_election.cc:341] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.854162 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:22.854852 775 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:22.856065 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.857003 891 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.857396 891 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.857704 891 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.851738 879 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.861541 845 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.862421 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:22.862807 845 ts_tablet_manager.cc:1397] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.126s user 0.015s sys 0.014s
I20260806 03:42:22.863034 775 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.862195 400 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:22.865610 845 raft_consensus.cc:348] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.867275 879 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.866353 845 raft_consensus.cc:374] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.873871 845 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.871681 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.874823 845 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.877463 842 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.877615 845 ts_tablet_manager.cc:1428] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.014s user 0.004s sys 0.003s
I20260806 03:42:22.877993 842 ts_tablet_manager.cc:1397] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.082s user 0.037s sys 0.018s
I20260806 03:42:22.878775 901 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.879048 845 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:22.879283 901 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.880064 843 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.880664 843 ts_tablet_manager.cc:1397] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.006s
I20260806 03:42:22.881522 901 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.881381 842 raft_consensus.cc:348] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.882105 842 raft_consensus.cc:374] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.882429 842 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.883085 842 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.883453 843 raft_consensus.cc:348] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.884452 843 raft_consensus.cc:374] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.884935 843 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.885411 842 ts_tablet_manager.cc:1428] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:42:22.885800 843 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.886404 845 tablet_bootstrap.cc:654] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.886294 842 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.888183 843 ts_tablet_manager.cc:1428] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:42:22.889089 843 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.892962 842 tablet_bootstrap.cc:654] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:22.894544 498 leader_election.cc:341] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.895311 843 tablet_bootstrap.cc:654] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.895848 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.896576 564 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.897677 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.898777 902 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.899163 902 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.899586 902 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.882078 901 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:22.903014 901 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.908814 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.910598 671 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.913071 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:22.923810 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:22.924786 674 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.925616 902 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.919991 901 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.929301 902 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.929136 891 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.930575 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.930751 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:22.930956 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:22.931352 674 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.931516 563 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.920349 854 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.936853 846 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:22.937268 846 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:22.937649 846 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.937978 842 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.938609 842 ts_tablet_manager.cc:1397] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.052s user 0.014s sys 0.004s
I20260806 03:42:22.939504 563 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:22.940750 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:22.942929 402 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:22.931466 564 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:22.941939 842 raft_consensus.cc:348] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.950766 891 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:22.951504 775 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:22.955094 842 raft_consensus.cc:374] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.955391 842 raft_consensus.cc:729] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.955590 501 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.952001 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:22.956084 842 consensus_queue.cc:260] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.956775 776 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.958092 842 ts_tablet_manager.cc:1428] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.019s user 0.003s sys 0.002s
I20260806 03:42:22.958971 842 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:22.953226 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.961109 854 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.961654 854 raft_consensus.cc:686] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:22.962546 854 consensus_queue.cc:237] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:22.965324 843 tablet_bootstrap.cc:492] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:22.965520 674 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:22.965848 843 ts_tablet_manager.cc:1397] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.077s user 0.030s sys 0.043s
I20260806 03:42:22.966130 902 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 03:42:22.966889 498 leader_election.cc:341] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:22.967511 902 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:22.968359 902 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.969462 843 raft_consensus.cc:348] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.970225 843 raft_consensus.cc:374] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.970602 843 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.971315 843 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.971748 846 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.973486 843 ts_tablet_manager.cc:1428] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:42:22.973706 846 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:22.974189 842 tablet_bootstrap.cc:654] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.974463 843 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:22.974666 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:22.975239 564 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.976609 776 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:22.977816 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:22.979183 854 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.979768 854 raft_consensus.cc:686] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:22.980780 854 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.983781 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:22.984658 674 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:22.954171 845 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:22.986449 845 ts_tablet_manager.cc:1397] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.108s user 0.029s sys 0.005s
I20260806 03:42:22.989027 843 tablet_bootstrap.cc:654] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:22.993306 842 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:22.993803 842 ts_tablet_manager.cc:1397] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.003s
I20260806 03:42:22.994994 402 catalog_manager.cc:5495] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:22.996318 842 raft_consensus.cc:348] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:22.997045 842 raft_consensus.cc:374] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:22.997376 842 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:22.997845 564 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:23.000725 845 raft_consensus.cc:348] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.001398 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:23.000005 674 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:23.002254 846 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:22.998031 842 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.003204 846 raft_consensus.cc:686] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:23.001425 845 raft_consensus.cc:374] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.004107 845 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.004057 846 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.005404 845 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.008281 845 ts_tablet_manager.cc:1428] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20260806 03:42:23.004140 842 ts_tablet_manager.cc:1428] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 03:42:23.005600 400 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:23.010102 842 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.017052 842 tablet_bootstrap.cc:654] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.019212 845 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.033025 845 tablet_bootstrap.cc:654] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.032109 401 catalog_manager.cc:5495] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:23.042295 855 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.043066 855 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.042295 846 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.046983 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.068177 776 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:23.067637 846 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.049760 891 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.070245 846 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.046890 855 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.070441 891 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.072091 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.072561 400 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:23.073873 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.075313 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.076143 671 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:23.049013 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.077469 614 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:23.078339 855 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.079905 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.080742 775 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 1: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:42:23.082274 501 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.083338 854 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.084340 727 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.085816 846 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.086313 855 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.093323 879 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.094019 846 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.094622 843 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.095008 843 ts_tablet_manager.cc:1397] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.121s user 0.021s sys 0.027s
I20260806 03:42:23.095705 891 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.104836 842 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.105319 842 ts_tablet_manager.cc:1397] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.095s user 0.017s sys 0.011s
I20260806 03:42:23.107752 854 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.097332 843 raft_consensus.cc:348] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.101792 855 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.107982 842 raft_consensus.cc:348] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.108711 842 raft_consensus.cc:374] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.108978 842 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.109697 843 raft_consensus.cc:374] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.109997 843 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.110129 842 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.110702 843 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.113057 843 ts_tablet_manager.cc:1428] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260806 03:42:23.114051 843 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.117980 842 ts_tablet_manager.cc:1428] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 03:42:23.121658 842 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
W20260806 03:42:23.124624 727 consensus_peers.cc:487] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:23.127389 843 tablet_bootstrap.cc:654] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.129134 855 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:23.131963 842 tablet_bootstrap.cc:654] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.134475 846 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.135365 846 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.141086 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.146176 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:23.147487 727 leader_election.cc:341] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:23.148319 727 leader_election.cc:341] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.149290 727 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.150167 902 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.138152 846 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.163170 885 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.163910 885 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.165692 889 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.167191 885 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.168497 885 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.169072 885 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.170045 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.172612 846 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.180222 846 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.182636 845 tablet_bootstrap.cc:492] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.182825 775 raft_consensus.cc:2379] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:23.184345 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.188877 614 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:23.189671 846 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.191370 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:23.192857 727 leader_election.cc:341] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.195842 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:23.197479 727 leader_election.cc:341] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.198047 727 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.199539 845 ts_tablet_manager.cc:1397] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.180s user 0.013s sys 0.030s
W20260806 03:42:23.205089 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:23.206084 846 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.214489 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.215237 775 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:23.222214 953 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.224080 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:23.224246 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.225063 563 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:23.229012 846 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:23.230854 855 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.231298 855 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.231633 855 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.235558 843 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.236081 843 ts_tablet_manager.cc:1397] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.122s user 0.020s sys 0.045s
I20260806 03:42:23.217170 885 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.244140 846 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.249471 843 raft_consensus.cc:348] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.250685 953 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.253569 842 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.257656 843 raft_consensus.cc:374] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.258131 843 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.258154 842 ts_tablet_manager.cc:1397] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.137s user 0.017s sys 0.045s
I20260806 03:42:23.259073 845 raft_consensus.cc:348] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.260073 845 raft_consensus.cc:374] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.260413 845 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.261224 845 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.265164 855 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.265961 842 raft_consensus.cc:348] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.267186 842 raft_consensus.cc:374] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.267716 842 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.268656 842 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.269492 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:23.270215 562 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.259745 843 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.272300 843 ts_tablet_manager.cc:1428] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.036s user 0.006s sys 0.002s
I20260806 03:42:23.275033 843 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.276410 842 ts_tablet_manager.cc:1428] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260806 03:42:23.277316 842 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.279971 845 ts_tablet_manager.cc:1428] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.079s user 0.005s sys 0.001s
I20260806 03:42:23.280957 845 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.287128 855 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.288153 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:23.288825 776 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.289793 842 tablet_bootstrap.cc:654] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.293958 562 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:23.294554 843 tablet_bootstrap.cc:654] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.295251 614 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:23.296594 855 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:23.297605 855 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:23.298514 855 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.304352 845 tablet_bootstrap.cc:654] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.305050 776 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:23.317088 891 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.317873 891 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.304260 889 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260806 03:42:23.323082 879 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.322829 891 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.329046 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.327653 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:23.331881 498 leader_election.cc:341] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.339876 846 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.330114 776 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:23.380923 854 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.389686 854 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.390676 854 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.394243 854 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.397637 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.398460 845 tablet_bootstrap.cc:492] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.399966 845 ts_tablet_manager.cc:1397] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.119s user 0.015s sys 0.029s
W20260806 03:42:23.400489 498 leader_election.cc:341] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.400560 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:23.402164 501 leader_election.cc:341] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.402645 501 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.400497 842 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.403721 845 raft_consensus.cc:348] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:23.404497 845 raft_consensus.cc:374] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.404670 842 ts_tablet_manager.cc:1397] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.128s user 0.060s sys 0.049s
I20260806 03:42:23.404790 845 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.405856 891 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.405683 845 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:42:23.408591 845 ts_tablet_manager.cc:1428] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:42:23.408589 842 raft_consensus.cc:348] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.409870 845 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.410020 842 raft_consensus.cc:374] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.410522 842 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.411217 842 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.413344 842 ts_tablet_manager.cc:1428] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 03:42:23.414575 842 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.416080 845 tablet_bootstrap.cc:654] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.423440 889 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.424961 501 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.428138 842 tablet_bootstrap.cc:654] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.428748 843 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.429292 843 ts_tablet_manager.cc:1397] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.154s user 0.006s sys 0.006s
I20260806 03:42:23.428462 889 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.431329 889 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.431708 902 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.432193 902 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.432761 843 raft_consensus.cc:348] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.433655 843 raft_consensus.cc:374] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.434011 843 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.433826 400 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:23.434720 843 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.435943 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:23.437464 727 leader_election.cc:341] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.434300 902 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.444078 902 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.431509 845 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.452171 845 ts_tablet_manager.cc:1397] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.004s
I20260806 03:42:23.431740 889 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.463503 845 raft_consensus.cc:348] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.463780 854 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:23.464319 845 raft_consensus.cc:374] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.464605 845 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.465255 845 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.465541 889 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.466439 671 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.467233 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.467829 889 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.468561 889 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.469096 889 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.471329 889 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.471652 845 ts_tablet_manager.cc:1428] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.019s user 0.002s sys 0.005s
I20260806 03:42:23.472563 845 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.473708 843 ts_tablet_manager.cc:1428] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260806 03:42:23.474627 843 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.479308 842 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.479777 842 ts_tablet_manager.cc:1397] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.065s user 0.026s sys 0.004s
I20260806 03:42:23.487771 901 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:42:23.487900 727 leader_election.cc:341] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.489197 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.490564 846 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.494932 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:23.495769 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.498188 842 raft_consensus.cc:348] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.498917 842 raft_consensus.cc:374] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.499203 842 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.499819 842 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.501827 842 ts_tablet_manager.cc:1428] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260806 03:42:23.504427 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:23.505028 776 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.506740 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:23.509207 501 leader_election.cc:341] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.503012 843 tablet_bootstrap.cc:654] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.512773 845 tablet_bootstrap.cc:654] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.516536 776 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:23.517858 501 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:23.518779 889 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:23.519469 889 raft_consensus.cc:686] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:23.520664 889 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.523588 901 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.526863 842 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
W20260806 03:42:23.531405 498 leader_election.cc:341] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:23.557518 498 leader_election.cc:341] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.558964 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.563517 891 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:42:23.561851 879 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.578751 842 tablet_bootstrap.cc:654] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.585812 845 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.586299 845 ts_tablet_manager.cc:1397] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.001s
I20260806 03:42:23.588743 902 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:23.590926 843 tablet_bootstrap.cc:492] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.591607 843 ts_tablet_manager.cc:1397] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.004s
I20260806 03:42:23.590896 400 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:23.594153 843 raft_consensus.cc:348] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.594907 843 raft_consensus.cc:374] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.595185 843 raft_consensus.cc:729] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.596068 845 raft_consensus.cc:348] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.596767 845 raft_consensus.cc:374] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.597026 845 raft_consensus.cc:729] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.598243 843 consensus_queue.cc:260] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.600512 843 ts_tablet_manager.cc:1428] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 03:42:23.601052 845 consensus_queue.cc:260] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.606359 845 ts_tablet_manager.cc:1428] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260806 03:42:23.609192 842 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.609740 842 ts_tablet_manager.cc:1397] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.019s
I20260806 03:42:23.611529 845 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.615126 843 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.618425 842 raft_consensus.cc:348] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.619944 842 raft_consensus.cc:374] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.620338 842 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.623416 845 tablet_bootstrap.cc:654] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.626112 842 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.634972 845 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.640960 845 ts_tablet_manager.cc:1397] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.007s
I20260806 03:42:23.643981 845 raft_consensus.cc:348] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.644745 845 raft_consensus.cc:374] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.645021 845 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.649106 842 ts_tablet_manager.cc:1428] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.039s user 0.010s sys 0.000s
I20260806 03:42:23.649871 843 tablet_bootstrap.cc:654] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.650570 842 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.645814 845 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.660908 845 ts_tablet_manager.cc:1428] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260806 03:42:23.662197 842 tablet_bootstrap.cc:654] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.664870 845 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.679173 843 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.679672 843 ts_tablet_manager.cc:1397] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.004s
I20260806 03:42:23.679764 845 tablet_bootstrap.cc:654] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.682767 843 raft_consensus.cc:348] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.683490 843 raft_consensus.cc:374] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.683786 843 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.685698 843 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.688885 843 ts_tablet_manager.cc:1428] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 03:42:23.689795 843 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.693784 842 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.694407 842 ts_tablet_manager.cc:1397] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.004s
I20260806 03:42:23.696305 843 tablet_bootstrap.cc:654] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.697544 842 raft_consensus.cc:348] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.698295 842 raft_consensus.cc:374] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.698702 842 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.699744 842 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.702970 842 ts_tablet_manager.cc:1428] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:42:23.704857 842 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.712463 842 tablet_bootstrap.cc:654] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.720126 902 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.720943 902 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.725481 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.730886 562 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:23.720057 855 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 03:42:23.738690 902 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.738600 727 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:23.740111 901 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.740197 846 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.740950 846 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.741585 846 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.741087 901 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.750891 845 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:23.751446 845 ts_tablet_manager.cc:1397] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.007s
I20260806 03:42:23.770905 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.772423 855 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.770982 891 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.773597 891 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.773268 855 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.793201 674 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:23.795058 727 leader_election.cc:341] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.796643 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.811038 889 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.815788 953 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.816519 953 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.832990 845 raft_consensus.cc:348] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.834821 845 raft_consensus.cc:374] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.835170 845 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.835927 845 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.811851 889 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.839963 985 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.839761 901 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.849651 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.850626 953 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.852887 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:23.853309 498 leader_election.cc:341] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.845409 985 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.856330 985 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.858750 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.861073 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.862013 671 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:23.846220 891 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.869132 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.853922 776 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:23.871805 843 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:23.872579 843 ts_tablet_manager.cc:1397] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.183s user 0.029s sys 0.058s
I20260806 03:42:23.869603 846 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.878834 845 ts_tablet_manager.cc:1428] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.127s user 0.015s sys 0.004s
W20260806 03:42:23.875718 501 leader_election.cc:341] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.881289 846 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.888216 845 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:23.859301 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:23.886921 727 leader_election.cc:341] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:23.887244 498 leader_election.cc:341] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:23.887370 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 69930.6us
I20260806 03:42:23.896276 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:23.899691 855 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.903421 773 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:23.915588 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.917138 891 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.917567 842 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:23.917711 891 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.918120 842 ts_tablet_manager.cc:1397] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.214s user 0.033s sys 0.044s
I20260806 03:42:23.918303 891 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.920415 843 raft_consensus.cc:348] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.921239 843 raft_consensus.cc:374] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.921490 843 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:23.921762 842 raft_consensus.cc:348] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.922482 842 raft_consensus.cc:374] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:23.913708 1003 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.911957 727 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:23.923282 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.923095 1003 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.923617 846 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.899657 889 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.924449 846 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.912959 498 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.925454 846 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.925931 1003 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.926610 842 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
W20260806 03:42:23.926983 498 leader_election.cc:341] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.922142 843 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.927266 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:23.927699 889 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.927451 842 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.888863 616 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:23.930689 562 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:23.931190 501 leader_election.cc:341] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:23.933127 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:23.934207 843 ts_tablet_manager.cc:1428] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260806 03:42:23.939389 985 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:23.941087 562 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:23.942911 843 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:23.945653 855 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:23.946558 855 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:23.947177 855 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.951794 842 ts_tablet_manager.cc:1428] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20260806 03:42:23.954460 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.954668 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.955245 564 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:23.954873 891 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.958874 891 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.960273 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:23.943806 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56823.4us
I20260806 03:42:23.963650 842 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:23.964313 1003 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:23.966076 727 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:23.966635 1003 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.967314 901 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:23.967888 901 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:23.969283 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:23.970789 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.969626 901 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.971879 773 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:23.970180 1003 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.974121 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:23.974907 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:23.976171 773 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.981114 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:23.981942 564 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:23.982481 846 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.985095 846 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:23.991202 855 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:23.993848 855 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:23.994529 616 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:23.997500 1006 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:23.997857 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:24.009102 842 tablet_bootstrap.cc:654] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:23.996791 614 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.009562 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.011827 616 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:24.012219 879 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.005376 773 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:24.015157 1003 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.011866 845 tablet_bootstrap.cc:654] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.015705 1003 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.016204 1003 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.015106 879 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.013077 843 tablet_bootstrap.cc:654] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.019209 879 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.022722 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:24.023314 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:24.023728 498 leader_election.cc:341] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.024015 889 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.024729 889 raft_consensus.cc:686] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:24.025266 953 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:24.025874 889 consensus_queue.cc:237] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.027218 727 leader_election.cc:341] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.027356 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.028327 776 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:24.023147 855 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:24.034112 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.034058 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:24.036777 562 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.036782 674 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.037034 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:24.045084 776 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:24.042898 501 leader_election.cc:341] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:24.045342 727 leader_election.cc:341] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.050499 402 catalog_manager.cc:5495] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:24.048218 400 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:24.060621 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.072402 674 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:24.069479 1003 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.077935 1003 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.079522 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:24.080320 773 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.081274 776 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:24.076432 1006 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:24.083396 616 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.085315 845 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:24.085922 845 ts_tablet_manager.cc:1397] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.199s user 0.015s sys 0.006s
I20260806 03:42:24.084623 1006 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.087107 1006 raft_consensus.cc:686] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:24.088132 1006 consensus_queue.cc:237] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.095726 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.096663 563 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.094280 902 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.099076 902 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.091785 498 leader_election.cc:341] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.102402 727 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:24.102843 498 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:24.105073 901 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.102283 902 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.109264 889 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:24.120622 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:24.122311 727 leader_election.cc:341] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.122486 773 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:24.125422 616 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.128674 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.130692 564 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:24.128105 1006 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.132194 1006 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:24.132128 889 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:24.134320 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:24.124794 901 raft_consensus.cc:686] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:24.124980 562 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:24.139060 1017 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.140198 842 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:24.140663 842 ts_tablet_manager.cc:1397] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.188s user 0.008s sys 0.015s
I20260806 03:42:24.139600 1017 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.141423 1017 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.144855 842 raft_consensus.cc:348] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.145401 842 raft_consensus.cc:374] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.145614 842 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.147186 563 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:24.147269 889 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:24.147539 901 consensus_queue.cc:237] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.146840 1006 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.151551 842 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.160929 843 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:24.170469 402 catalog_manager.cc:5495] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:24.265990 843 ts_tablet_manager.cc:1397] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.323s user 0.031s sys 0.039s
I20260806 03:42:24.163818 842 ts_tablet_manager.cc:1428] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20260806 03:42:24.264532 845 raft_consensus.cc:348] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.273649 842 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:24.273799 845 raft_consensus.cc:374] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.274106 845 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.226745 400 catalog_manager.cc:5495] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:24.274801 845 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.265995 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 109721us
I20260806 03:42:24.283944 845 ts_tablet_manager.cc:1428] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.198s user 0.006s sys 0.004s
I20260806 03:42:24.284621 845 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:24.294013 843 raft_consensus.cc:348] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.294792 843 raft_consensus.cc:374] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.295074 843 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.295694 843 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.296241 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:24.298581 843 ts_tablet_manager.cc:1428] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260806 03:42:24.299715 843 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:24.313174 845 tablet_bootstrap.cc:654] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.315066 843 tablet_bootstrap.cc:654] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.307883 1017 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.316948 842 tablet_bootstrap.cc:654] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.326057 401 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:24.330493 1017 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.335131 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.336059 562 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.335995 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:24.342249 727 leader_election.cc:341] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.354179 845 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:24.354592 562 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:24.355247 845 ts_tablet_manager.cc:1397] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.011s
I20260806 03:42:24.356163 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:24.358577 1017 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.359198 842 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:24.361090 842 ts_tablet_manager.cc:1397] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.088s user 0.013s sys 0.008s
I20260806 03:42:24.360152 845 raft_consensus.cc:348] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.362592 845 raft_consensus.cc:374] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.362959 845 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.359236 1017 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:24.364866 1017 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.366111 845 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.368330 842 raft_consensus.cc:348] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.369338 842 raft_consensus.cc:374] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.369822 842 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.373041 842 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.375020 845 ts_tablet_manager.cc:1428] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260806 03:42:24.375900 845 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:24.376350 842 ts_tablet_manager.cc:1428] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260806 03:42:24.379551 842 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:24.382014 843 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:24.383443 843 ts_tablet_manager.cc:1397] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.009s
I20260806 03:42:24.390900 843 raft_consensus.cc:348] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.391606 843 raft_consensus.cc:374] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.391839 843 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.392692 843 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.393338 842 tablet_bootstrap.cc:654] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.394871 843 ts_tablet_manager.cc:1428] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260806 03:42:24.395731 843 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:24.396443 400 catalog_manager.cc:5495] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:24.402490 843 tablet_bootstrap.cc:654] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.406734 842 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:24.407310 842 ts_tablet_manager.cc:1397] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.016s
I20260806 03:42:24.412583 842 raft_consensus.cc:348] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.413424 842 raft_consensus.cc:374] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.413695 842 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.413786 843 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:24.414328 843 ts_tablet_manager.cc:1397] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.011s
I20260806 03:42:24.414674 842 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.417091 845 tablet_bootstrap.cc:654] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.418062 1006 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.418630 1006 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.421808 1006 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.422926 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.422973 843 raft_consensus.cc:348] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.424070 843 raft_consensus.cc:374] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.424099 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.424486 843 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.422183 985 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:42:24.425554 614 leader_election.cc:341] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.425524 843 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.425696 985 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.427299 616 leader_election.cc:341] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.429479 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:24.431193 498 leader_election.cc:341] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.432551 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:24.433784 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.434283 1006 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:24.434630 773 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:24.436134 501 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:24.437214 843 ts_tablet_manager.cc:1428] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260806 03:42:24.437243 889 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.437934 889 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.438352 889 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.438747 843 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:24.442600 985 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.447839 842 ts_tablet_manager.cc:1428] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.040s user 0.001s sys 0.008s
I20260806 03:42:24.449293 889 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.450115 843 tablet_bootstrap.cc:654] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.453548 842 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:24.455753 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:24.457698 498 leader_election.cc:341] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.462900 889 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.463506 845 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:24.464042 845 ts_tablet_manager.cc:1397] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.036s
I20260806 03:42:24.464305 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:24.464877 773 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.467310 842 tablet_bootstrap.cc:654] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.467156 845 raft_consensus.cc:348] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.467947 845 raft_consensus.cc:374] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.468228 845 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.469065 845 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.471879 845 ts_tablet_manager.cc:1428] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:42:24.471940 901 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.474963 845 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:24.474416 901 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.474763 1017 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:24.477564 901 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.480492 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.486806 563 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:24.502991 1006 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.504513 1006 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.511564 845 tablet_bootstrap.cc:654] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.516185 727 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:24.518422 901 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.520054 901 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.523196 901 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.523025 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.525993 843 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:24.526574 843 ts_tablet_manager.cc:1397] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.011s
I20260806 03:42:24.529701 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.530714 563 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:24.532055 614 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.533166 1003 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.533651 1003 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.534178 1003 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.535523 1006 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.536566 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.536240 843 raft_consensus.cc:348] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.534092 773 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:24.537569 843 raft_consensus.cc:374] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.538645 843 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.539326 843 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.539554 616 leader_election.cc:341] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.541795 843 ts_tablet_manager.cc:1428] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 03:42:24.544597 842 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:24.545066 842 ts_tablet_manager.cc:1397] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.035s
I20260806 03:42:24.546770 843 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
W20260806 03:42:24.550798 498 consensus_peers.cc:487] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:24.553340 501 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:42:24.553923 727 leader_election.cc:341] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.555092 985 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.555727 985 raft_consensus.cc:686] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:24.557096 985 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.560905 674 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.562323 901 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.584682 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.590804 562 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.593317 1003 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.597738 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:24.598745 563 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.612854 1006 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:24.635572 562 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:24.651739 879 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:42:24.664093 901 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.665294 845 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:24.665088 889 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.666558 845 ts_tablet_manager.cc:1397] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.192s user 0.011s sys 0.015s
I20260806 03:42:24.668493 842 raft_consensus.cc:348] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.670118 842 raft_consensus.cc:374] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.670425 842 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.671274 842 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.665877 889 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.675958 842 ts_tablet_manager.cc:1428] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.131s user 0.007s sys 0.003s
I20260806 03:42:24.676998 842 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:24.676913 845 raft_consensus.cc:348] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.677700 845 raft_consensus.cc:374] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.677989 845 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.678488 563 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:24.678609 845 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.679776 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.680554 773 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:24.681435 889 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.683419 501 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:24.686120 889 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.687678 901 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:24.687711 889 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.688323 727 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:24.688678 889 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.689508 1003 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.690196 1017 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.691016 1017 raft_consensus.cc:686] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:24.692265 1017 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.700140 614 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.702133 843 tablet_bootstrap.cc:654] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.706950 1040 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.707607 1040 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.708834 889 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.710898 1040 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.711618 1040 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:24.712262 1040 raft_consensus.cc:686] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:24.713270 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:24.714087 776 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.716464 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.713266 1040 consensus_queue.cc:237] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.733341 727 consensus_peers.cc:487] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Couldn't send request to peer ada5ef6656cb4420baf77a4b4342909e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:24.734392 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:24.734602 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.735106 671 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:24.736500 846 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.737167 846 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.740561 846 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.741783 879 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.753206 845 ts_tablet_manager.cc:1428] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.086s user 0.010s sys 0.015s
I20260806 03:42:24.753535 879 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.754987 845 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:24.756462 401 catalog_manager.cc:5495] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:24.711704 889 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.763137 1039 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260806 03:42:24.766746 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:24.763747 1042 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 03:42:24.778193 842 tablet_bootstrap.cc:654] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.759933 846 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:24.782585 846 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.783790 879 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:24.780393 671 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:24.794652 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.788022 776 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:24.807890 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:24.800768 400 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:24.857641 843 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:24.858143 843 ts_tablet_manager.cc:1397] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.312s user 0.007s sys 0.044s
W20260806 03:42:24.822093 616 leader_election.cc:341] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.851708 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.808892 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.822611 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:24.849189 846 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:24.820729 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.807898 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.871758 668 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:24.884212 845 tablet_bootstrap.cc:654] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:24.887048 985 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
W20260806 03:42:24.886857 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 79108.7us
I20260806 03:42:24.888206 985 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:24.889710 985 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.892727 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 82537.2us
W20260806 03:42:24.898185 727 leader_election.cc:341] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.901269 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:24.902117 562 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:24.873497 616 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:24.908133 843 raft_consensus.cc:348] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:24.909247 843 raft_consensus.cc:374] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:24.909591 843 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:24.912732 843 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:24.917033 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48034.4us
I20260806 03:42:24.942456 843 ts_tablet_manager.cc:1428] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.084s user 0.009s sys 0.000s
I20260806 03:42:24.943456 843 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:24.935201 498 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:24.946543 879 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:24.947003 879 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:24.947379 879 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:24.948961 727 leader_election.cc:341] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:24.949985 727 leader_election.cc:341] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:24.954840 501 leader_election.cc:341] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:24.959100 727 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:24.966532 1042 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:24.965823 400 catalog_manager.cc:5495] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:24.978207 846 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:24.992547 842 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:24.993197 842 ts_tablet_manager.cc:1397] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.316s user 0.023s sys 0.011s
I20260806 03:42:24.994450 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:24.961956 901 consensus_queue.cc:1035] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260806 03:42:24.999297 842 raft_consensus.cc:348] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.002712 842 raft_consensus.cc:374] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.003139 842 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.004083 842 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.007972 901 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:25.010119 727 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:24.947278 1040 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.040s
I20260806 03:42:24.956061 879 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.020864 901 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:25.025312 901 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:25.025774 901 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:25.031481 727 consensus_peers.cc:487] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:42:25.022243 1066 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 03:42:25.035164 843 tablet_bootstrap.cc:654] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.039844 842 ts_tablet_manager.cc:1428] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.046s user 0.010s sys 0.000s
I20260806 03:42:25.046769 879 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:25.047178 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:25.046825 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:25.048753 669 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:25.048874 727 leader_election.cc:341] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.051653 845 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.052573 845 ts_tablet_manager.cc:1397] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.298s user 0.033s sys 0.018s
I20260806 03:42:25.052693 842 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
W20260806 03:42:25.056233 501 leader_election.cc:341] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.054495 1017 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:25.057780 1017 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.072774 1017 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:25.064138 901 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.081043 901 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:25.081776 842 tablet_bootstrap.cc:654] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.083320 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:25.083879 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:25.084898 563 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.085127 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:25.088449 669 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:25.113621 498 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:25.116974 563 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:25.123917 402 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:25.126886 845 raft_consensus.cc:348] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.128328 1073 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:25.133551 1073 raft_consensus.cc:686] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:25.135197 1073 consensus_queue.cc:237] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:25.129642 727 leader_election.cc:341] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.131213 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:25.138474 845 raft_consensus.cc:374] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.139050 845 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.152926 843 tablet_bootstrap.cc:492] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:25.153916 843 ts_tablet_manager.cc:1397] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.211s user 0.010s sys 0.007s
W20260806 03:42:25.156525 498 consensus_peers.cc:487] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:25.165727 843 raft_consensus.cc:348] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.166471 843 raft_consensus.cc:374] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.166752 843 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.162681 845 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.167352 843 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.163265 727 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:25.171872 901 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:25.178556 901 raft_consensus.cc:686] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:25.179978 901 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:25.198860 727 leader_election.cc:341] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.199950 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:25.227545 1017 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:25.235467 727 leader_election.cc:341] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.240237 842 tablet_bootstrap.cc:492] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:25.239262 1073 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.240787 842 ts_tablet_manager.cc:1397] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.188s user 0.030s sys 0.012s
I20260806 03:42:25.242096 845 ts_tablet_manager.cc:1428] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.189s user 0.008s sys 0.003s
I20260806 03:42:25.241683 843 ts_tablet_manager.cc:1428] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.087s user 0.007s sys 0.000s
I20260806 03:42:25.245033 843 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:25.250049 845 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:25.261332 401 catalog_manager.cc:5495] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:25.259794 842 raft_consensus.cc:348] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.263976 842 raft_consensus.cc:374] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.264248 842 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.265419 842 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.275313 1003 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.296729 1003 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:25.297438 1003 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.300419 1003 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:25.318651 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:25.326334 842 ts_tablet_manager.cc:1428] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.085s user 0.010s sys 0.000s
I20260806 03:42:25.327410 842 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:25.329972 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:25.337124 616 consensus_peers.cc:487] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 -> Peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Couldn't send request to peer 0374a84d5e9f4e3c8d8287e2382379c8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:25.343302 1082 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.344175 616 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:25.345494 1003 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:25.381585 843 tablet_bootstrap.cc:654] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.382566 845 tablet_bootstrap.cc:654] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.381464 401 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:25.403908 879 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260806 03:42:25.418087 879 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.432562 842 tablet_bootstrap.cc:654] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:25.452069 727 consensus_peers.cc:487] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:25.464771 879 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:42:25.463635 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:25.467082 879 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.493881 843 tablet_bootstrap.cc:492] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:25.495136 843 ts_tablet_manager.cc:1397] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.250s user 0.050s sys 0.016s
I20260806 03:42:25.498623 879 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:25.500958 845 tablet_bootstrap.cc:492] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.501367 845 ts_tablet_manager.cc:1397] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.252s user 0.018s sys 0.048s
I20260806 03:42:25.507001 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:25.509404 498 leader_election.cc:341] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.519742 845 raft_consensus.cc:348] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.520732 845 raft_consensus.cc:374] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.521073 845 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.519030 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:25.530437 772 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:25.533664 845 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.541887 501 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:25.543988 1073 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:25.544380 1073 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:25.544749 1073 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.549619 843 raft_consensus.cc:348] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.550398 843 raft_consensus.cc:374] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.550648 843 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.551250 843 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.553318 843 ts_tablet_manager.cc:1428] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.057s user 0.003s sys 0.003s
I20260806 03:42:25.554520 843 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:25.558884 563 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.559587 674 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.599501 775 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.601270 845 ts_tablet_manager.cc:1428] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.100s user 0.010s sys 0.008s
I20260806 03:42:25.602021 845 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:25.602576 1073 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.605664 1073 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:25.606683 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:25.607000 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:25.607754 772 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:25.609191 498 leader_election.cc:341] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.615255 842 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:25.615732 842 ts_tablet_manager.cc:1397] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.289s user 0.046s sys 0.029s
I20260806 03:42:25.625525 1073 consensus_queue.cc:1035] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.627497 842 raft_consensus.cc:348] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.628167 842 raft_consensus.cc:374] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.628420 842 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.629192 842 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.631271 842 ts_tablet_manager.cc:1428] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
I20260806 03:42:25.632159 842 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:25.636613 843 tablet_bootstrap.cc:654] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.642757 772 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:25.647372 845 tablet_bootstrap.cc:654] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.648247 842 tablet_bootstrap.cc:654] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.650211 501 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:25.654762 879 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:25.655494 879 raft_consensus.cc:686] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:25.664851 772 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.657130 879 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.679426 845 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.680223 845 ts_tablet_manager.cc:1397] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.010s
I20260806 03:42:25.683655 845 raft_consensus.cc:348] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.684572 845 raft_consensus.cc:374] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.684895 845 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.685817 845 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.689041 845 ts_tablet_manager.cc:1428] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 03:42:25.692608 845 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:25.700855 845 tablet_bootstrap.cc:654] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.725687 845 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.726531 845 ts_tablet_manager.cc:1397] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.006s
I20260806 03:42:25.736690 985 consensus_queue.cc:1035] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.736136 400 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:25.729959 845 raft_consensus.cc:348] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.753861 845 raft_consensus.cc:374] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.754585 845 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.756060 845 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.759855 843 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:25.760710 843 ts_tablet_manager.cc:1397] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.206s user 0.015s sys 0.074s
I20260806 03:42:25.761853 845 ts_tablet_manager.cc:1428] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20260806 03:42:25.763631 845 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:25.769795 669 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.779320 842 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:25.780159 842 ts_tablet_manager.cc:1397] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.034s
I20260806 03:42:25.783464 843 raft_consensus.cc:348] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.786741 843 raft_consensus.cc:374] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.787174 843 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.788154 843 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.790799 843 ts_tablet_manager.cc:1428] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.030s user 0.009s sys 0.000s
I20260806 03:42:25.798061 843 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:25.817548 901 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.822691 668 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:25.824545 845 tablet_bootstrap.cc:654] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.844939 843 tablet_bootstrap.cc:654] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.853266 842 raft_consensus.cc:348] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.854578 842 raft_consensus.cc:374] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.855023 842 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.854903 901 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:25.855995 842 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.858438 1082 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:25.859665 842 ts_tablet_manager.cc:1428] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.079s user 0.009s sys 0.004s
I20260806 03:42:25.855952 901 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.860813 842 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:25.862286 901 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:25.864194 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:25.869590 727 leader_election.cc:341] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.869930 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:25.872153 727 leader_election.cc:341] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:25.872742 727 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:25.874166 901 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:25.883695 845 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.884208 845 ts_tablet_manager.cc:1397] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.121s user 0.016s sys 0.012s
I20260806 03:42:25.884487 1006 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:25.885288 1006 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.894570 1006 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:25.896759 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:25.897424 845 raft_consensus.cc:348] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.898151 845 raft_consensus.cc:374] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.898432 845 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.899055 845 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.899456 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:25.900496 842 tablet_bootstrap.cc:654] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.904708 614 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:25.907601 1006 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:25.909978 845 ts_tablet_manager.cc:1428] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.026s user 0.007s sys 0.002s
I20260806 03:42:25.911295 845 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:25.920830 843 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:25.921368 843 ts_tablet_manager.cc:1397] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.124s user 0.018s sys 0.011s
I20260806 03:42:25.927578 845 tablet_bootstrap.cc:654] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.927153 843 raft_consensus.cc:348] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.927924 843 raft_consensus.cc:374] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.928177 843 raft_consensus.cc:729] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.928645 842 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:25.929083 842 ts_tablet_manager.cc:1397] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.021s
I20260806 03:42:25.928778 843 consensus_queue.cc:260] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.930706 843 ts_tablet_manager.cc:1428] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260806 03:42:25.931762 843 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:25.932363 842 raft_consensus.cc:348] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.941592 842 raft_consensus.cc:374] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.942001 842 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.942745 842 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.945518 842 ts_tablet_manager.cc:1428] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260806 03:42:25.947237 842 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:25.954227 843 tablet_bootstrap.cc:654] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.958348 842 tablet_bootstrap.cc:654] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:25.987579 843 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:25.988415 843 ts_tablet_manager.cc:1397] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.057s user 0.014s sys 0.017s
I20260806 03:42:25.991829 845 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:25.992985 845 ts_tablet_manager.cc:1397] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.082s user 0.016s sys 0.027s
I20260806 03:42:25.993194 843 raft_consensus.cc:348] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.994979 843 raft_consensus.cc:374] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:25.995247 843 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:25.995869 843 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:25.998144 843 ts_tablet_manager.cc:1428] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.003s sys 0.004s
I20260806 03:42:25.999100 843 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.001092 845 raft_consensus.cc:348] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.002085 845 raft_consensus.cc:374] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.002437 845 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.004631 845 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.007376 842 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:26.007496 845 ts_tablet_manager.cc:1428] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260806 03:42:26.008087 842 ts_tablet_manager.cc:1397] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.020s
I20260806 03:42:26.010195 845 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:26.011088 843 tablet_bootstrap.cc:654] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.011615 842 raft_consensus.cc:348] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.012339 842 raft_consensus.cc:374] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.012590 842 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.013502 842 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.018563 845 tablet_bootstrap.cc:654] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.020341 842 ts_tablet_manager.cc:1428] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 03:42:26.021641 842 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:26.027565 843 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.028059 843 ts_tablet_manager.cc:1397] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.011s
I20260806 03:42:26.031136 843 raft_consensus.cc:348] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.031862 843 raft_consensus.cc:374] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.032102 843 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.032648 843 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.033320 902 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.034018 902 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.035089 843 ts_tablet_manager.cc:1428] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:42:26.035956 843 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.037154 842 tablet_bootstrap.cc:654] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.037243 902 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.041247 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:26.043087 727 leader_election.cc:341] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.045310 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:26.049444 727 leader_election.cc:341] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.049939 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.050122 843 tablet_bootstrap.cc:654] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.050709 901 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:26.053021 674 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.073303 845 tablet_bootstrap.cc:492] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:26.073676 842 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:26.073768 845 ts_tablet_manager.cc:1397] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.065s user 0.013s sys 0.010s
I20260806 03:42:26.072479 879 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.074224 842 ts_tablet_manager.cc:1397] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.014s
I20260806 03:42:26.074463 879 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.078691 843 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.079454 843 ts_tablet_manager.cc:1397] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.044s user 0.018s sys 0.008s
I20260806 03:42:26.080986 879 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.083774 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.084990 773 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:26.093914 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:26.100211 879 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.100935 879 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.101434 879 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.126968 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.129156 843 raft_consensus.cc:348] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.132800 843 raft_consensus.cc:374] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.133204 843 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.134024 843 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.129447 845 raft_consensus.cc:348] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:26.130813 498 leader_election.cc:341] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.137410 843 ts_tablet_manager.cc:1428] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.058s user 0.008s sys 0.007s
I20260806 03:42:26.139070 843 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.139496 845 raft_consensus.cc:374] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.139798 845 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.148277 843 tablet_bootstrap.cc:654] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.133594 842 raft_consensus.cc:348] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.155382 842 raft_consensus.cc:374] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.155673 842 raft_consensus.cc:729] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.156267 842 consensus_queue.cc:260] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.140444 845 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.165629 845 ts_tablet_manager.cc:1428] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.091s user 0.009s sys 0.006s
I20260806 03:42:26.168838 842 ts_tablet_manager.cc:1428] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.094s user 0.004s sys 0.004s
I20260806 03:42:26.170069 842 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:26.174188 845 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:26.176003 879 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.181272 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:26.181898 773 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.182497 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.182924 879 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:26.184825 498 leader_election.cc:341] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.187996 842 tablet_bootstrap.cc:654] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.192351 843 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.193069 843 ts_tablet_manager.cc:1397] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.011s
I20260806 03:42:26.193323 845 tablet_bootstrap.cc:654] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.196810 843 raft_consensus.cc:348] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.197680 843 raft_consensus.cc:374] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.197978 843 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.198704 843 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.202554 879 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.192421 773 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:26.203310 879 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.205928 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.208585 1110 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.209306 1110 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:42:26.212234 498 consensus_peers.cc:487] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:26.215821 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.215660 1110 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.216984 773 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:26.218434 501 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:26.220139 1073 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.220741 1073 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.221300 1073 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.221500 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.214430 879 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.222370 671 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:26.232069 843 ts_tablet_manager.cc:1428] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.039s user 0.005s sys 0.003s
I20260806 03:42:26.232430 1073 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.234993 843 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.249259 845 tablet_bootstrap.cc:492] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:26.249714 842 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:26.249779 845 ts_tablet_manager.cc:1397] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.076s user 0.013s sys 0.017s
I20260806 03:42:26.250730 842 ts_tablet_manager.cc:1397] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.017s
I20260806 03:42:26.256155 1073 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.256690 845 raft_consensus.cc:348] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.257524 845 raft_consensus.cc:374] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.257795 845 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.258407 845 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.261637 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:26.261895 842 raft_consensus.cc:348] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.262253 775 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.261528 402 catalog_manager.cc:5495] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:26.264036 842 raft_consensus.cc:374] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.264359 842 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.265638 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.266585 845 ts_tablet_manager.cc:1428] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260806 03:42:26.265137 842 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.267719 845 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:26.267084 843 tablet_bootstrap.cc:654] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.267892 671 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.272778 842 ts_tablet_manager.cc:1428] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20260806 03:42:26.273669 1110 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:42:26.277587 1073 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.284365 1073 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.287573 1073 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.297777 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.290074 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.298861 773 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
W20260806 03:42:26.299921 498 leader_election.cc:341] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.304471 842 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:26.308627 501 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.313411 1110 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.313949 1110 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.314401 1110 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.293725 671 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:26.317359 498 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:26.318533 879 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.319130 879 raft_consensus.cc:686] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:26.320204 879 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.323704 845 tablet_bootstrap.cc:654] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.310340 775 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:26.346457 1073 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.349677 1073 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.365741 1073 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.367069 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.368290 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:26.369575 498 leader_election.cc:341] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.339108 1110 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.386245 843 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.388252 843 ts_tablet_manager.cc:1397] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.153s user 0.030s sys 0.019s
W20260806 03:42:26.384735 501 leader_election.cc:341] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.396701 843 raft_consensus.cc:348] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.397156 1110 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.397469 843 raft_consensus.cc:374] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.397775 843 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.397948 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.398438 843 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:26.399127 498 leader_election.cc:341] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.400389 843 ts_tablet_manager.cc:1428] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260806 03:42:26.401173 843 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.393438 501 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.407506 1110 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:26.411556 669 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.417696 400 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:26.422259 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:26.421293 879 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.423372 773 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.423498 879 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.427920 843 tablet_bootstrap.cc:654] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.435817 845 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:26.436491 845 ts_tablet_manager.cc:1397] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.169s user 0.018s sys 0.033s
I20260806 03:42:26.446218 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.449877 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.436581 879 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.450922 772 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:26.455016 501 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:26.456605 1110 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.457195 1110 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.457614 1110 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.459335 842 tablet_bootstrap.cc:654] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.454980 845 raft_consensus.cc:348] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.445542 773 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:26.465024 845 raft_consensus.cc:374] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.469347 845 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.470077 845 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:26.473977 498 leader_election.cc:341] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.474728 1048 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.475409 1048 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.479650 1048 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.475701 501 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.482142 879 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.482955 879 raft_consensus.cc:686] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:26.484622 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.484344 879 consensus_queue.cc:237] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.485736 901 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.486240 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:26.489387 901 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:26.490806 614 leader_election.cc:341] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.494612 845 ts_tablet_manager.cc:1428] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20260806 03:42:26.495815 901 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.494503 773 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:26.497645 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.498296 674 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:26.500111 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:26.501591 1040 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.502002 1040 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.502405 1040 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.503880 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:26.507684 727 leader_election.cc:341] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.509352 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:26.514415 901 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.510202 843 tablet_bootstrap.cc:492] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.515755 843 ts_tablet_manager.cc:1397] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.009s
I20260806 03:42:26.514950 901 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.516450 1048 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.522037 843 raft_consensus.cc:348] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.526285 902 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.517365 901 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.535159 843 raft_consensus.cc:374] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.538453 843 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.539019 902 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.551246 902 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.552464 1006 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:42:26.557349 1006 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.564975 845 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:26.569301 843 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.539372 1048 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.539417 1110 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.540359 1040 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.587262 843 ts_tablet_manager.cc:1428] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.071s user 0.013s sys 0.000s
I20260806 03:42:26.588665 843 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.592396 1006 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.581898 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.598395 1048 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.588579 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:26.601714 400 catalog_manager.cc:5495] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:26.605968 842 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:26.606534 842 ts_tablet_manager.cc:1397] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.302s user 0.007s sys 0.012s
W20260806 03:42:26.610251 727 leader_election.cc:341] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.611827 1110 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.616443 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.616494 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:26.618525 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:26.620107 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:26.620651 498 leader_election.cc:341] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.621320 775 raft_consensus.cc:2379] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:26.614712 564 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:26.629772 842 raft_consensus.cc:348] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.630733 842 raft_consensus.cc:374] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.631053 842 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.632436 901 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.631867 842 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.640997 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:26.642692 727 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.644991 902 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.645591 902 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.646257 902 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.655018 842 ts_tablet_manager.cc:1428] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.048s user 0.008s sys 0.000s
I20260806 03:42:26.644061 775 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:42:26.656953 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:26.658735 1151 log.cc:924] Time spent T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 03:42:26.660871 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:26.664676 671 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.665642 845 tablet_bootstrap.cc:654] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.667325 842 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:26.644451 1040 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:26.682942 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:26.683830 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.686139 1155 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.698081 1048 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:26.698571 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:26.699767 669 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 1: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:42:26.700943 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:26.701864 772 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.696332 901 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.706503 1155 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.708377 902 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.711299 902 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:42:26.713678 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:26.711735 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:26.721998 842 tablet_bootstrap.cc:654] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.721748 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.724056 843 tablet_bootstrap.cc:654] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.715080 1155 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.728312 669 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:26.735004 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:26.735810 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:26.736860 614 leader_election.cc:341] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.737376 614 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:26.738302 1040 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260806 03:42:26.738515 727 leader_election.cc:341] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.739308 1157 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:26.740204 1157 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:26.741856 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:26.742064 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:26.742750 561 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:26.745324 727 leader_election.cc:341] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.745772 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.746845 902 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:42:26.742763 562 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:26.749670 845 tablet_bootstrap.cc:492] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:26.750083 845 ts_tablet_manager.cc:1397] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.191s user 0.024s sys 0.008s
I20260806 03:42:26.738345 772 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:26.741912 1157 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.754984 501 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.755962 1048 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:26.756096 1073 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.756630 1073 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:26.757468 1073 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.760746 843 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.761523 843 ts_tablet_manager.cc:1397] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.173s user 0.009s sys 0.014s
I20260806 03:42:26.763734 845 raft_consensus.cc:348] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.764472 845 raft_consensus.cc:374] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.764778 845 raft_consensus.cc:729] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.768257 845 consensus_queue.cc:260] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.770445 843 raft_consensus.cc:348] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.771183 843 raft_consensus.cc:374] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.771441 843 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.773049 845 ts_tablet_manager.cc:1428] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260806 03:42:26.773059 1157 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.774397 845 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:26.780673 842 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:26.781265 842 ts_tablet_manager.cc:1397] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.010s
I20260806 03:42:26.776467 843 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.785496 879 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:26.788208 1157 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.789255 843 ts_tablet_manager.cc:1428] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20260806 03:42:26.790191 843 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:26.791108 842 raft_consensus.cc:348] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.791903 842 raft_consensus.cc:374] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.792225 842 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.793081 842 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.795782 842 ts_tablet_manager.cc:1428] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20260806 03:42:26.797025 842 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:26.797099 561 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:26.801309 843 tablet_bootstrap.cc:654] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.801106 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:26.802421 669 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.812275 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:26.813392 562 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:26.823678 727 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.823791 1110 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:26.825070 1157 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.826036 1157 raft_consensus.cc:686] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:26.827562 1157 consensus_queue.cc:237] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.851336 845 tablet_bootstrap.cc:654] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:26.855209 669 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:26.861019 400 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:26.864428 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:26.868898 901 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:26.870045 901 raft_consensus.cc:686] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:26.875135 1155 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:26.876098 1155 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.877856 901 consensus_queue.cc:237] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.882616 1155 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:26.905804 843 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:26.906523 843 ts_tablet_manager.cc:1397] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.117s user 0.050s sys 0.040s
I20260806 03:42:26.921446 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:26.925747 842 tablet_bootstrap.cc:654] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:26.931463 727 leader_election.cc:341] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.934705 879 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:26.936957 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:26.939608 562 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
W20260806 03:42:26.947954 727 leader_election.cc:341] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:26.948462 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:26.951921 846 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:26.960127 1110 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:26.983825 879 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:26.994743 843 raft_consensus.cc:348] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:26.995620 843 raft_consensus.cc:374] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:26.995888 843 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:26.998292 400 catalog_manager.cc:5495] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:26.996449 843 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.003551 843 ts_tablet_manager.cc:1428] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.097s user 0.020s sys 0.003s
I20260806 03:42:27.004637 843 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:27.042136 671 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.045387 842 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:27.046216 842 ts_tablet_manager.cc:1397] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.249s user 0.013s sys 0.023s
I20260806 03:42:27.067646 985 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.073594 985 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.076813 985 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.083609 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:27.084583 771 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:27.086346 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.087997 501 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:27.089469 845 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:27.090179 845 ts_tablet_manager.cc:1397] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.316s user 0.008s sys 0.027s
W20260806 03:42:27.091022 498 leader_election.cc:341] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.094823 401 catalog_manager.cc:5495] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:27.096949 845 raft_consensus.cc:348] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.097641 845 raft_consensus.cc:374] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.097872 845 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.102305 902 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.098623 845 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.094259 985 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:27.104637 985 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:27.104983 985 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.103910 902 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.131769 902 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.135133 842 raft_consensus.cc:348] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.136133 842 raft_consensus.cc:374] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.136538 842 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.133724 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:27.138072 842 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.138228 1073 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.139145 1073 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.141302 842 ts_tablet_manager.cc:1428] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.095s user 0.008s sys 0.000s
I20260806 03:42:27.147075 842 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:27.152882 845 ts_tablet_manager.cc:1428] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.062s user 0.009s sys 0.000s
I20260806 03:42:27.153813 1073 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.155951 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.157107 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:27.158582 845 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:27.159890 843 tablet_bootstrap.cc:654] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:27.161947 727 leader_election.cc:341] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:27.163321 501 leader_election.cc:341] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:27.167057 727 leader_election.cc:341] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.167685 727 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.169247 902 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.173725 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.194705 902 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.194809 879 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:27.194192 985 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.195590 902 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.200341 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:27.201131 771 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.201799 902 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.201023 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.203231 1073 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:42:27.204893 727 leader_election.cc:341] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:27.206025 498 leader_election.cc:341] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.206842 498 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.214512 842 tablet_bootstrap.cc:654] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.222676 1073 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.224639 985 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.219736 771 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:27.228735 501 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:27.229622 891 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.230305 891 raft_consensus.cc:686] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:27.231665 845 tablet_bootstrap.cc:654] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.231644 891 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.235010 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:27.242128 843 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:27.242976 843 ts_tablet_manager.cc:1397] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.239s user 0.011s sys 0.019s
I20260806 03:42:27.253398 669 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.256475 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:27.296967 498 leader_election.cc:341] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:27.302600 727 leader_election.cc:341] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.303043 727 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.306958 843 raft_consensus.cc:348] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.308164 843 raft_consensus.cc:374] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.308545 843 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.301885 1110 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.318218 1110 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.304837 901 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.323027 843 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.325459 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:27.326368 843 ts_tablet_manager.cc:1428] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.083s user 0.005s sys 0.004s
I20260806 03:42:27.327915 843 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:27.328541 400 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:27.321301 1110 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:27.332067 501 leader_election.cc:341] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:27.343007 498 consensus_peers.cc:487] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:27.384862 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.388919 842 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:27.389510 842 ts_tablet_manager.cc:1397] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.247s user 0.046s sys 0.073s
I20260806 03:42:27.393132 842 raft_consensus.cc:348] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.406543 842 raft_consensus.cc:374] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.407076 842 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.408223 842 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.410270 842 ts_tablet_manager.cc:1428] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.020s user 0.001s sys 0.005s
I20260806 03:42:27.398114 901 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:27.399987 902 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:27.402359 843 tablet_bootstrap.cc:654] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.403534 845 tablet_bootstrap.cc:492] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:27.414597 845 ts_tablet_manager.cc:1397] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.256s user 0.018s sys 0.022s
I20260806 03:42:27.417018 842 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
W20260806 03:42:27.430246 498 leader_election.cc:341] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.434087 498 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.421037 845 raft_consensus.cc:348] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.437461 845 raft_consensus.cc:374] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.437908 845 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.440088 845 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.439461 879 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.495071 843 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:27.495831 843 ts_tablet_manager.cc:1397] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.168s user 0.021s sys 0.003s
I20260806 03:42:27.496964 845 ts_tablet_manager.cc:1428] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.082s user 0.009s sys 0.000s
I20260806 03:42:27.497962 845 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:27.508558 671 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.518150 843 raft_consensus.cc:348] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.516101 902 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 03:42:27.519201 843 raft_consensus.cc:374] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.519534 843 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.520771 842 tablet_bootstrap.cc:654] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.520305 843 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.522730 1040 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:42:27.523787 1040 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.548278 843 ts_tablet_manager.cc:1428] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.052s user 0.008s sys 0.000s
I20260806 03:42:27.550483 843 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:27.554931 1040 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.556704 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:27.558472 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:27.569288 616 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:27.572669 1040 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:27.573963 879 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.574715 879 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.581192 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:27.582702 498 leader_election.cc:341] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.584069 879 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.593551 845 tablet_bootstrap.cc:654] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.592947 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:27.597893 501 leader_election.cc:341] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.597908 842 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:27.598327 501 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.599062 842 ts_tablet_manager.cc:1397] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.182s user 0.013s sys 0.015s
I20260806 03:42:27.603135 1155 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:27.601415 879 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.616083 902 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.616688 902 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.620170 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:27.621873 727 leader_election.cc:341] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.622550 842 raft_consensus.cc:348] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.623061 902 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.624933 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:27.626482 727 leader_election.cc:341] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.626843 727 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.628341 902 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:27.623386 842 raft_consensus.cc:374] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.630438 842 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.632135 842 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.634297 842 ts_tablet_manager.cc:1428] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.035s user 0.002s sys 0.003s
I20260806 03:42:27.635397 842 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:27.644088 842 tablet_bootstrap.cc:654] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.648703 843 tablet_bootstrap.cc:654] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.648576 902 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.649480 902 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.652381 902 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.657553 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:27.663739 727 leader_election.cc:341] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.694164 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:27.696348 564 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:27.698516 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.699990 902 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:27.700697 902 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:27.701538 902 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.709854 845 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:27.710454 845 ts_tablet_manager.cc:1397] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.213s user 0.019s sys 0.036s
I20260806 03:42:27.776412 845 raft_consensus.cc:348] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.777506 845 raft_consensus.cc:374] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.777971 845 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.779773 879 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.780771 879 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.785910 879 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.794037 845 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.797519 845 ts_tablet_manager.cc:1428] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.087s user 0.022s sys 0.038s
I20260806 03:42:27.798509 845 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:27.810974 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.819895 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:27.821141 772 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:27.824463 842 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:27.825152 842 ts_tablet_manager.cc:1397] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.190s user 0.004s sys 0.031s
I20260806 03:42:27.830585 845 tablet_bootstrap.cc:654] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.832020 501 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:42:27.832535 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:27.829751 842 raft_consensus.cc:348] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.838096 842 raft_consensus.cc:374] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.838179 879 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:27.838454 842 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.838649 879 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:27.839073 879 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.793390 902 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.839568 842 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.841629 842 ts_tablet_manager.cc:1428] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.016s user 0.000s sys 0.010s
I20260806 03:42:27.842469 842 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:27.844732 1073 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:27.851387 845 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:27.851845 845 ts_tablet_manager.cc:1397] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.003s
I20260806 03:42:27.852752 879 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.855998 879 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:27.858632 902 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.859274 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:27.863090 772 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.863036 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:27.865196 561 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:27.864398 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:27.872718 843 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:27.873416 843 ts_tablet_manager.cc:1397] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.323s user 0.013s sys 0.008s
W20260806 03:42:27.879719 498 leader_election.cc:341] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.881402 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:27.883687 845 raft_consensus.cc:348] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:27.884195 727 leader_election.cc:341] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.884420 845 raft_consensus.cc:374] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.884745 845 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.885442 845 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.891160 843 raft_consensus.cc:348] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.891755 843 raft_consensus.cc:374] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:27.892074 843 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:27.892894 843 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.896960 845 ts_tablet_manager.cc:1428] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20260806 03:42:27.898574 845 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
W20260806 03:42:27.901036 498 consensus_peers.cc:487] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:42:27.901594 498 leader_election.cc:341] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:27.910552 561 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:27.913203 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.914609 902 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.915488 902 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:27.917435 902 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.924749 843 ts_tablet_manager.cc:1428] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.051s user 0.008s sys 0.004s
I20260806 03:42:27.925812 843 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:27.926334 842 tablet_bootstrap.cc:654] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.936964 772 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:27.939306 501 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:27.940858 1073 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:27.941835 1073 raft_consensus.cc:686] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:27.949097 1073 consensus_queue.cc:237] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.953130 843 tablet_bootstrap.cc:654] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.968591 845 tablet_bootstrap.cc:654] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:27.978765 400 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:27.984826 901 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:27.985621 843 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:27.986135 843 ts_tablet_manager.cc:1397] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.018s
I20260806 03:42:27.985616 901 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:27.990008 901 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:27.991492 842 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:27.991958 842 ts_tablet_manager.cc:1397] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.150s user 0.020s sys 0.019s
I20260806 03:42:27.993110 401 catalog_manager.cc:5495] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:27.994758 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:27.996733 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:27.998601 843 raft_consensus.cc:348] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:28.000559 727 leader_election.cc:341] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.000795 843 raft_consensus.cc:374] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.001194 843 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
W20260806 03:42:28.001499 727 leader_election.cc:341] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.001721 843 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.002013 727 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.007465 902 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.008617 843 ts_tablet_manager.cc:1428] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20260806 03:42:28.009534 843 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.011457 842 raft_consensus.cc:348] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.012297 842 raft_consensus.cc:374] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.012566 842 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.013195 842 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.013736 902 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.015506 842 ts_tablet_manager.cc:1428] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260806 03:42:28.015585 902 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.016455 842 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:28.017648 902 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.018570 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.020156 843 tablet_bootstrap.cc:654] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:28.022265 727 leader_election.cc:341] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.021067 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:28.025442 727 leader_election.cc:341] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.028518 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.029255 842 tablet_bootstrap.cc:654] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.030145 902 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.040541 1073 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.041179 1073 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.053100 1073 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.054991 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.056113 668 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 1: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:42:28.057574 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.058463 775 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 1: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:42:28.059548 501 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.060734 1073 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.064906 845 tablet_bootstrap.cc:492] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:28.065433 845 ts_tablet_manager.cc:1397] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.167s user 0.015s sys 0.020s
I20260806 03:42:28.065760 843 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:28.066201 843 ts_tablet_manager.cc:1397] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.004s
I20260806 03:42:28.067972 845 raft_consensus.cc:348] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.068681 845 raft_consensus.cc:374] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.069032 845 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.068720 843 raft_consensus.cc:348] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.069607 843 raft_consensus.cc:374] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.069938 843 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.069660 845 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.070581 1073 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.070703 843 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.072211 845 ts_tablet_manager.cc:1428] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:42:28.073263 845 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:28.075067 843 ts_tablet_manager.cc:1428] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:42:28.076076 842 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:28.076094 843 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.076712 842 ts_tablet_manager.cc:1397] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.015s
I20260806 03:42:28.081051 842 raft_consensus.cc:348] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.081575 842 raft_consensus.cc:374] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.081800 842 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.085146 842 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.087216 845 tablet_bootstrap.cc:654] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.088306 842 ts_tablet_manager.cc:1428] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260806 03:42:28.089239 842 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:28.089354 843 tablet_bootstrap.cc:654] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.092445 1187 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.093025 1187 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.095218 1187 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.098145 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.109411 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.110265 775 raft_consensus.cc:2379] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:28.111698 616 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:28.113061 1040 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.114691 1187 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.115185 1187 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.117208 1187 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.119172 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.120751 1040 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:28.121222 614 leader_election.cc:341] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.123974 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.124821 772 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:28.126538 616 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:28.127163 842 tablet_bootstrap.cc:654] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.130024 1040 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.130519 1040 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.130909 1040 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.132405 843 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:28.132853 843 ts_tablet_manager.cc:1397] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.019s
I20260806 03:42:28.141870 843 raft_consensus.cc:348] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.142342 845 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:28.142666 843 raft_consensus.cc:374] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.142956 843 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.143090 845 ts_tablet_manager.cc:1397] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.024s
I20260806 03:42:28.143838 1040 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.143893 843 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.146708 843 ts_tablet_manager.cc:1428] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260806 03:42:28.146663 1040 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.148262 843 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.148229 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:28.148056 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:28.148962 775 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:28.149801 614 leader_election.cc:341] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.154150 842 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:28.157491 842 ts_tablet_manager.cc:1397] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.006s
I20260806 03:42:28.160802 775 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:28.162494 616 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:28.164088 1040 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.172107 842 raft_consensus.cc:348] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.172647 843 tablet_bootstrap.cc:654] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.172628 1040 raft_consensus.cc:686] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:28.173048 842 raft_consensus.cc:374] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.173382 842 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.173710 1040 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.173777 845 raft_consensus.cc:348] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.174034 842 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.175621 845 raft_consensus.cc:374] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.176096 845 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.178481 845 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.180927 845 ts_tablet_manager.cc:1428] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.038s user 0.007s sys 0.001s
I20260806 03:42:28.181887 845 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:28.177822 842 ts_tablet_manager.cc:1428] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.020s user 0.000s sys 0.017s
I20260806 03:42:28.183326 842 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:28.209985 843 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:28.210565 843 ts_tablet_manager.cc:1397] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.063s user 0.022s sys 0.019s
I20260806 03:42:28.212510 842 tablet_bootstrap.cc:654] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.213033 843 raft_consensus.cc:348] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.215368 843 raft_consensus.cc:374] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.215667 843 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.217625 843 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.230799 843 ts_tablet_manager.cc:1428] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260806 03:42:28.227104 401 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:28.258935 843 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.253566 842 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:28.260825 842 ts_tablet_manager.cc:1397] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.078s user 0.022s sys 0.023s
I20260806 03:42:28.219061 845 tablet_bootstrap.cc:654] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.264853 842 raft_consensus.cc:348] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.265636 842 raft_consensus.cc:374] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.265904 842 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.267408 842 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.270570 842 ts_tablet_manager.cc:1428] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260806 03:42:28.271859 842 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:28.276775 843 tablet_bootstrap.cc:654] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.287074 902 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.299328 902 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.300946 842 tablet_bootstrap.cc:654] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.304380 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.305119 564 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.307698 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.308617 668 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.313323 727 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:28.315289 1155 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.315951 1155 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.316515 1155 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.342686 1040 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.302043 902 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.350169 1073 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.351091 1073 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.348495 1040 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.353996 1073 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.356776 902 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:28.357914 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.359408 561 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:42:28.360787 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:28.362656 501 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.361897 1040 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.368877 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.372550 616 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:28.373368 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.373802 1040 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:28.376474 498 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.376986 498 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.378463 985 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.395058 843 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:28.395769 843 ts_tablet_manager.cc:1397] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.137s user 0.041s sys 0.059s
I20260806 03:42:28.397126 845 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:28.397540 845 ts_tablet_manager.cc:1397] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.216s user 0.033s sys 0.032s
I20260806 03:42:28.404059 843 raft_consensus.cc:348] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.404798 843 raft_consensus.cc:374] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.405054 843 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.405598 843 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.407007 845 raft_consensus.cc:348] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.407667 845 raft_consensus.cc:374] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.407922 845 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.408172 843 ts_tablet_manager.cc:1428] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260806 03:42:28.409225 843 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.414036 845 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.415850 1157 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.416848 1157 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.417969 842 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:28.418586 842 ts_tablet_manager.cc:1397] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.147s user 0.030s sys 0.030s
I20260806 03:42:28.413515 1155 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.422789 1155 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.425267 1157 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.418272 845 ts_tablet_manager.cc:1428] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.020s user 0.003s sys 0.007s
I20260806 03:42:28.431130 1236 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.431463 845 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:28.431870 1236 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:28.438886 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 49312.3us
I20260806 03:42:28.440397 843 tablet_bootstrap.cc:654] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.441672 985 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:28.443378 1236 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.447929 1040 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.455479 845 tablet_bootstrap.cc:654] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.460521 842 raft_consensus.cc:348] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.461372 842 raft_consensus.cc:374] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.461642 842 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.462057 985 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.462901 985 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.462410 842 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.465278 842 ts_tablet_manager.cc:1428] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.046s user 0.007s sys 0.001s
I20260806 03:42:28.465687 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.466234 842 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
W20260806 03:42:28.467249 498 leader_election.cc:341] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.469462 985 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.474442 1073 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.474929 1073 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.478899 1073 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.480891 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.483587 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.486261 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:28.486472 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:28.487133 667 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.487855 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.488070 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:28.490986 501 leader_election.cc:341] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.490068 1040 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.492425 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.492810 563 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.493547 561 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.492501 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.492659 1040 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.497442 564 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.497789 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.501650 772 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:28.503311 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.506594 1073 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.507055 1073 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.507545 1073 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.543490 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:28.545711 1187 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.547137 901 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.547629 901 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.547991 901 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.546420 1187 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:28.553117 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 49888.6us
I20260806 03:42:28.557364 727 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:28.583033 843 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:28.583545 843 ts_tablet_manager.cc:1397] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.175s user 0.012s sys 0.047s
I20260806 03:42:28.585668 1187 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.574349 1155 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.591890 842 tablet_bootstrap.cc:654] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.593684 843 raft_consensus.cc:348] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.594521 843 raft_consensus.cc:374] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.594871 843 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.595676 843 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.584059 1040 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.613981 843 ts_tablet_manager.cc:1428] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.030s user 0.006s sys 0.003s
I20260806 03:42:28.591167 1155 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.632242 1155 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:28.651232 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 84120.7us
I20260806 03:42:28.658814 843 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:28.666154 845 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:28.666738 845 ts_tablet_manager.cc:1397] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.236s user 0.024s sys 0.038s
W20260806 03:42:28.667625 727 leader_election.cc:341] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.649191 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:28.668797 727 leader_election.cc:341] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:28.670480 498 leader_election.cc:341] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.680603 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.681842 1247 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.683704 901 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.690002 901 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.674134 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.669150 1157 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 03:42:28.679292 561 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:28.694756 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.717080 564 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:28.693903 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:28.695621 902 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.723989 727 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:28.725250 901 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.725739 901 raft_consensus.cc:686] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:28.729463 614 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:28.726862 901 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.700189 772 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:28.743465 845 raft_consensus.cc:348] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.744289 845 raft_consensus.cc:374] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.744587 845 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.743222 1187 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:42:28.745949 1187 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.746237 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:28.746585 1187 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:28.749028 845 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.727029 902 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.759486 845 ts_tablet_manager.cc:1428] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.092s user 0.007s sys 0.001s
I20260806 03:42:28.760711 845 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:28.762181 616 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:28.755440 667 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:28.766309 1255 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:28.778054 901 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:28.770004 902 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.778607 901 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.782876 1245 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:28.787253 901 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.794615 1073 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.798430 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.802866 843 tablet_bootstrap.cc:654] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:28.798309 1073 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:28.815361 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:28.816627 770 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.824124 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:28.825790 562 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:28.826152 727 leader_election.cc:341] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.826274 1155 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.828905 1155 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:42:28.831694 727 leader_election.cc:341] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:28.837177 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.838022 558 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.917348 842 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:28.919763 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.921732 842 ts_tablet_manager.cc:1397] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.456s user 0.021s sys 0.024s
I20260806 03:42:28.934326 1236 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.948833 1236 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.900991 1252 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.105s
I20260806 03:42:28.869810 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:28.957896 546 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:28.960767 842 raft_consensus.cc:348] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.961548 842 raft_consensus.cc:374] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:28.961826 842 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:28.962877 727 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:28.964954 901 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:28.965317 901 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:28.965667 901 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.962494 842 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.901101 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:28.952685 1236 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.924896 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:42:28.894586 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 112113us
I20260806 03:42:28.931979 562 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:28.970808 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:28.971592 842 ts_tablet_manager.cc:1428] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20260806 03:42:28.977926 402 catalog_manager.cc:5495] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:28.979030 842 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:28.979599 901 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:28.985883 901 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:28.987954 668 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:28.988448 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:28.992628 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:28.992818 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:28.993379 548 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:28.994143 1155 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:28.994697 1155 raft_consensus.cc:686] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:28.995517 1155 consensus_queue.cc:237] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.007047 843 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:29.007565 843 ts_tablet_manager.cc:1397] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.349s user 0.003s sys 0.045s
I20260806 03:42:28.990518 558 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.011865 843 raft_consensus.cc:348] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.012398 843 raft_consensus.cc:374] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.012614 843 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.013147 843 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.016861 1187 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.018939 1187 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:29.014703 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:29.003383 614 consensus_peers.cc:487] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 -> Peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Couldn't send request to peer ada5ef6656cb4420baf77a4b4342909e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:42:29.024085 498 leader_election.cc:341] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:29.025364 727 leader_election.cc:341] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.030906 845 tablet_bootstrap.cc:654] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.031894 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:29.032207 770 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:29.032539 771 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:29.035226 843 ts_tablet_manager.cc:1428] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260806 03:42:29.039289 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:29.046331 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.050798 1073 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:29.051352 1073 raft_consensus.cc:686] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:42:29.076630 727 leader_election.cc:341] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.077721 548 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:29.052713 1073 consensus_queue.cc:237] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:29.078641 727 leader_election.cc:341] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.054039 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:29.085704 843 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:29.046121 771 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:29.089807 562 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:29.096964 1255 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:29.104104 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:29.112234 727 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.117286 901 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:29.117769 901 raft_consensus.cc:686] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:29.118527 901 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.128573 1236 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.103060 1255 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:29.171386 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 57982.7us
I20260806 03:42:29.198827 842 tablet_bootstrap.cc:654] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.205615 1260 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:29.206257 1260 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:29.206459 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:29.207304 546 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.208047 1255 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:29.215400 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:29.216447 776 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:29.218168 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:29.219058 1245 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 03:42:29.219226 1268 log.cc:924] Time spent T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.054s user 0.000s sys 0.001s
I20260806 03:42:29.219653 1245 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:29.220161 1245 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.224347 562 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:29.225914 845 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:29.226423 845 ts_tablet_manager.cc:1397] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.466s user 0.028s sys 0.049s
I20260806 03:42:29.231273 845 raft_consensus.cc:348] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.231827 845 raft_consensus.cc:374] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.232165 845 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.232786 845 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.240765 546 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:29.241283 843 tablet_bootstrap.cc:654] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.246500 558 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:29.207787 1260 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.222611 1236 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:29.248854 401 catalog_manager.cc:5495] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:29.252241 845 ts_tablet_manager.cc:1428] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.026s user 0.005s sys 0.002s
W20260806 03:42:29.252947 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 53050.2us
I20260806 03:42:29.254731 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:29.254307 845 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
W20260806 03:42:29.264671 727 leader_election.cc:341] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.270906 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.274507 901 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:29.275153 901 raft_consensus.cc:686] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:29.277786 727 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.276147 901 consensus_queue.cc:237] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.284401 1236 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:29.287667 1236 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.295979 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:29.297705 1236 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:29.300529 561 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.288230 1250 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:29.310197 1250 raft_consensus.cc:686] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:29.311718 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:29.319802 1250 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.331508 665 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.335335 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:29.340408 1236 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:29.342296 400 catalog_manager.cc:5495] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 2, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:29.352555 727 leader_election.cc:341] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:29.354902 727 leader_election.cc:341] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.355322 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.358518 1155 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:29.359187 614 leader_election.cc:341] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.308089 1245 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:29.462234 1283 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260806 03:42:29.476310 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 72532.6us
I20260806 03:42:29.494288 842 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:29.494926 842 ts_tablet_manager.cc:1397] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.516s user 0.024s sys 0.059s
I20260806 03:42:29.495819 902 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 03:42:29.506682 843 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:29.507318 843 ts_tablet_manager.cc:1397] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.422s user 0.073s sys 0.095s
I20260806 03:42:29.507830 1245 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:29.516435 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:29.519723 842 raft_consensus.cc:348] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.520579 842 raft_consensus.cc:374] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.520869 842 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.521605 842 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.535980 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:29.517387 773 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.537492 843 raft_consensus.cc:348] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.538264 843 raft_consensus.cc:374] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.539109 842 ts_tablet_manager.cc:1428] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.044s user 0.006s sys 0.007s
I20260806 03:42:29.538653 843 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.539989 843 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.541607 842 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:29.550700 843 ts_tablet_manager.cc:1428] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.043s user 0.008s sys 0.001s
I20260806 03:42:29.552187 845 tablet_bootstrap.cc:654] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.552538 843 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:29.572012 562 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:29.557109 1155 consensus_queue.cc:1035] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 03:42:29.607450 773 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:29.609467 843 tablet_bootstrap.cc:654] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.626294 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:29.630846 1245 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:29.631803 1245 raft_consensus.cc:686] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:29.635079 842 tablet_bootstrap.cc:654] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:29.622795 1250 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:29.633380 1245 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.644367 1250 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.654033 1250 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:42:29.662233 727 consensus_peers.cc:487] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:29.661433 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:29.663322 1279 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260806 03:42:29.667500 727 leader_election.cc:341] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.671535 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:29.682138 727 leader_election.cc:341] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:29.682602 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:29.683395 1250 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:29.684572 727 consensus_peers.cc:487] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:29.684710 562 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:29.692785 1260 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:42:29.706797 1260 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.788800 564 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:42:29.801043 843 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:29.801636 843 ts_tablet_manager.cc:1397] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.249s user 0.009s sys 0.012s
I20260806 03:42:29.823309 845 tablet_bootstrap.cc:492] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:29.823959 845 ts_tablet_manager.cc:1397] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.570s user 0.017s sys 0.025s
I20260806 03:42:29.829205 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:29.812700 1245 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260806 03:42:29.835683 843 raft_consensus.cc:348] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.836673 843 raft_consensus.cc:374] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.837005 843 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.837803 843 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.814086 1236 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
I20260806 03:42:29.842481 1260 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:29.880344 1299 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.180s user 0.000s sys 0.002s
W20260806 03:42:29.907754 727 consensus_peers.cc:487] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:29.908221 843 ts_tablet_manager.cc:1428] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.102s user 0.013s sys 0.001s
I20260806 03:42:29.886660 1292 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20260806 03:42:29.973133 843 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:29.975787 773 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:42:29.980207 845 raft_consensus.cc:348] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.981287 845 raft_consensus.cc:374] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:29.981706 845 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:29.983594 845 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:29.984292 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:29.989395 1304 log.cc:924] Time spent T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
W20260806 03:42:30.053364 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 121417us
I20260806 03:42:30.055246 902 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.056510 902 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.122274 845 ts_tablet_manager.cc:1428] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.235s user 0.007s sys 0.002s
I20260806 03:42:30.115527 1307 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:42:30.116516 1250 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.123615 1250 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.122453 902 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:42:30.124379 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61432.5us
I20260806 03:42:30.128017 1250 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:30.128083 616 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:30.130551 1040 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:30.137472 843 tablet_bootstrap.cc:654] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.144480 845 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:30.144712 401 catalog_manager.cc:5495] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:30.151032 401 catalog_manager.cc:5495] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
W20260806 03:42:30.161947 498 consensus_peers.cc:487] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:30.198868 845 tablet_bootstrap.cc:654] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.185420 401 catalog_manager.cc:5495] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:30.201921 1308 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.204911 1308 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.205680 842 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:30.206306 671 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.203150 1307 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:30.205699 401 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:30.214638 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.231855 1308 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:30.246649 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.256402 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:30.258682 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:30.272109 842 ts_tablet_manager.cc:1397] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.731s user 0.020s sys 0.015s
W20260806 03:42:30.300184 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:30.313170 1256 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:42:30.323758 727 leader_election.cc:341] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:30.325150 727 leader_election.cc:341] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.336324 845 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:30.337037 845 ts_tablet_manager.cc:1397] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.193s user 0.017s sys 0.010s
I20260806 03:42:30.337675 843 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:30.338965 843 ts_tablet_manager.cc:1397] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.366s user 0.013s sys 0.021s
I20260806 03:42:30.341765 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:30.343741 843 raft_consensus.cc:348] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.344678 843 raft_consensus.cc:374] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.345027 843 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.345815 843 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.380088 843 ts_tablet_manager.cc:1428] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.041s user 0.005s sys 0.004s
I20260806 03:42:30.381779 845 raft_consensus.cc:348] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.382747 845 raft_consensus.cc:374] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.383074 845 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.384765 845 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.387270 842 raft_consensus.cc:348] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.388321 842 raft_consensus.cc:374] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.388602 842 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.389338 842 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.392205 843 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:30.408385 845 ts_tablet_manager.cc:1428] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.071s user 0.006s sys 0.006s
I20260806 03:42:30.410905 845 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:30.445613 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:30.446184 843 tablet_bootstrap.cc:654] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.453408 842 ts_tablet_manager.cc:1428] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.172s user 0.002s sys 0.010s
W20260806 03:42:30.448614 727 consensus_peers.cc:487] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:42:30.458716 727 leader_election.cc:341] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.463420 727 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:30.465332 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:30.465862 1306 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:30.469379 1040 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:30.471665 727 leader_election.cc:341] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.472137 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:30.476599 1256 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:30.479192 842 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:30.477310 902 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:30.545764 1252 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.561554 1252 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.564826 845 tablet_bootstrap.cc:654] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:30.571854 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 73255.4us
W20260806 03:42:30.580859 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 80228.5us
W20260806 03:42:30.666340 1341 log.cc:924] Time spent T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.102s user 0.000s sys 0.005s
I20260806 03:42:30.609992 661 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.673555 842 tablet_bootstrap.cc:654] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.673806 1236 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.676304 1236 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.686524 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.697805 1252 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:30.700770 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 103880us
W20260806 03:42:30.702654 498 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.685554 1236 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:30.711935 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.722831 1353 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.723515 1353 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.731892 843 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:30.732630 843 ts_tablet_manager.cc:1397] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.341s user 0.027s sys 0.007s
I20260806 03:42:30.748672 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:30.760859 727 leader_election.cc:341] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.763576 842 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:30.764148 842 ts_tablet_manager.cc:1397] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.285s user 0.011s sys 0.014s
I20260806 03:42:30.764343 845 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:30.764787 845 ts_tablet_manager.cc:1397] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.354s user 0.035s sys 0.028s
I20260806 03:42:30.766389 402 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:30.769575 727 leader_election.cc:341] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.771803 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:30.779357 1353 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:30.770030 727 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:30.780548 1236 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:30.783262 842 raft_consensus.cc:348] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.784024 842 raft_consensus.cc:374] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.784283 842 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.785640 843 raft_consensus.cc:348] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.786329 843 raft_consensus.cc:374] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.786630 843 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.787281 843 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:30.787920 501 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.788367 501 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:30.789441 1110 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:30.790678 842 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.795010 842 ts_tablet_manager.cc:1428] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.031s user 0.002s sys 0.004s
I20260806 03:42:30.795567 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.796036 842 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:30.790347 845 raft_consensus.cc:348] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.798454 845 raft_consensus.cc:374] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.798775 845 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.799523 845 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.799913 1315 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.803965 668 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:30.805657 843 ts_tablet_manager.cc:1428] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.073s user 0.008s sys 0.000s
I20260806 03:42:30.806780 843 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:30.809039 845 ts_tablet_manager.cc:1428] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260806 03:42:30.810075 845 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:30.811124 498 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:30.800379 1315 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.813481 1252 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:30.814392 1252 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:30.814713 1315 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:30.814744 1252 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.817435 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:30.821847 727 leader_election.cc:341] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.822945 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:30.812106 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:30.824570 769 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
W20260806 03:42:30.827767 727 leader_election.cc:341] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:30.828193 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:30.828883 1236 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:30.829094 843 tablet_bootstrap.cc:654] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.831545 1252 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.833467 842 tablet_bootstrap.cc:654] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.834702 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:30.835773 663 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.841795 1252 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:30.848349 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:30.849074 769 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:30.856781 845 tablet_bootstrap.cc:654] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.859186 1187 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:42:30.859809 1187 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.864548 1187 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:30.865751 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:30.868395 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:30.870986 614 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:30.873983 1187 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:30.876122 663 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:30.877816 769 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:30.882360 498 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:30.883268 1252 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:30.884194 1252 raft_consensus.cc:686] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:30.885208 1252 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.895516 843 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:30.897696 843 ts_tablet_manager.cc:1397] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.091s user 0.015s sys 0.007s
I20260806 03:42:30.901254 843 raft_consensus.cc:348] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.902796 843 raft_consensus.cc:374] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.903206 843 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.904021 843 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.907398 843 ts_tablet_manager.cc:1428] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.004s sys 0.003s
I20260806 03:42:30.907755 845 tablet_bootstrap.cc:492] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:30.908232 845 ts_tablet_manager.cc:1397] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.002s
I20260806 03:42:30.908680 843 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:30.910876 845 raft_consensus.cc:348] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.911619 845 raft_consensus.cc:374] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.911883 845 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.915537 843 tablet_bootstrap.cc:654] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.936416 845 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.942356 842 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:30.942947 842 ts_tablet_manager.cc:1397] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.147s user 0.008s sys 0.009s
I20260806 03:42:30.945708 842 raft_consensus.cc:348] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.946952 845 ts_tablet_manager.cc:1428] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.038s user 0.009s sys 0.000s
I20260806 03:42:30.948010 845 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:30.952610 842 raft_consensus.cc:374] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:30.953083 842 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:30.953915 842 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.959998 400 catalog_manager.cc:5495] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:30.970039 842 ts_tablet_manager.cc:1428] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260806 03:42:30.973397 845 tablet_bootstrap.cc:654] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:30.976598 842 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:30.982940 843 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:30.991981 1252 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:30.992555 1252 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:30.996526 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:30.997160 661 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:30.997479 1252 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:30.998224 498 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:31.000208 842 tablet_bootstrap.cc:654] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.000836 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.001511 769 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:31.003530 1252 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.004400 1252 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.004757 1252 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.008836 843 ts_tablet_manager.cc:1397] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.100s user 0.015s sys 0.027s
I20260806 03:42:31.016101 1252 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.019375 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.019580 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:31.020042 661 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.020247 769 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.020412 1110 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.020907 1110 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.022961 1110 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.024598 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.025285 772 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:31.026451 501 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:31.026628 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.027313 1110 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.027743 1110 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:42:31.027993 498 leader_election.cc:341] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.028156 1110 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.030658 769 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:31.031862 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:31.032105 661 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:31.032665 1252 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.032766 1253 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.033289 1253 raft_consensus.cc:686] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:31.019404 843 raft_consensus.cc:348] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.033957 1253 consensus_queue.cc:237] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.040145 843 raft_consensus.cc:374] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.042119 843 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.045075 843 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.047806 1110 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.049799 1110 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.052949 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.055164 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:31.055721 769 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:31.058615 498 leader_election.cc:341] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.063151 842 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:31.063665 842 ts_tablet_manager.cc:1397] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.002s
I20260806 03:42:31.064288 769 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:31.065863 501 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.067101 1252 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.068010 1252 raft_consensus.cc:686] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:31.068945 1252 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.066341 842 raft_consensus.cc:348] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.073413 842 raft_consensus.cc:374] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.074131 1315 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.074649 1315 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.075551 843 ts_tablet_manager.cc:1428] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.066s user 0.007s sys 0.001s
I20260806 03:42:31.076761 1315 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.078869 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.080173 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:31.080310 727 leader_election.cc:341] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:31.081849 727 leader_election.cc:341] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.082255 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.084079 1256 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.087128 845 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.085753 401 catalog_manager.cc:5495] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:31.092690 845 ts_tablet_manager.cc:1397] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.145s user 0.019s sys 0.008s
I20260806 03:42:31.098994 843 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.102942 845 raft_consensus.cc:348] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.103739 845 raft_consensus.cc:374] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.104008 845 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.104646 845 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.073741 842 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.106707 845 ts_tablet_manager.cc:1428] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260806 03:42:31.108088 845 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.110987 842 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.114414 842 ts_tablet_manager.cc:1428] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20260806 03:42:31.119813 842 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:31.123838 843 tablet_bootstrap.cc:654] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.135232 1236 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.135967 1236 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.137966 1236 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.140048 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.141211 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:31.142536 727 leader_election.cc:341] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.141911 845 tablet_bootstrap.cc:654] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:31.143728 727 leader_election.cc:341] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.144148 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.144860 1256 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.149245 845 tablet_bootstrap.cc:492] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.149379 403 catalog_manager.cc:5495] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:31.149964 845 ts_tablet_manager.cc:1397] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.002s
I20260806 03:42:31.153712 842 tablet_bootstrap.cc:654] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.154534 845 raft_consensus.cc:348] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.155252 845 raft_consensus.cc:374] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.155576 845 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.160378 1256 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.160854 1256 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.163103 1256 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.166663 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.167599 562 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:31.169965 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:31.172488 727 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.174410 843 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.174871 843 ts_tablet_manager.cc:1397] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.012s
I20260806 03:42:31.177279 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.176873 845 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.178457 1236 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.178828 1236 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.179312 1236 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.179597 845 ts_tablet_manager.cc:1428] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20260806 03:42:31.179327 843 raft_consensus.cc:348] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.180371 843 raft_consensus.cc:374] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.180603 843 raft_consensus.cc:729] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.183106 843 consensus_queue.cc:260] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.195470 843 ts_tablet_manager.cc:1428] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260806 03:42:31.199352 845 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.202184 843 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.210292 843 tablet_bootstrap.cc:654] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.205253 1236 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.221400 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:31.222011 562 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.227553 845 tablet_bootstrap.cc:654] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.228223 842 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:31.228698 842 ts_tablet_manager.cc:1397] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.109s user 0.018s sys 0.019s
I20260806 03:42:31.229462 562 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:31.230505 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:31.231254 1256 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.231169 842 raft_consensus.cc:348] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.231768 1256 raft_consensus.cc:686] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:31.231981 842 raft_consensus.cc:374] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.232280 842 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.232614 1256 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.232879 842 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.235109 842 ts_tablet_manager.cc:1428] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 03:42:31.239660 842 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:31.240327 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:31.242412 727 leader_election.cc:341] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.242969 843 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.243391 1236 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.243597 843 ts_tablet_manager.cc:1397] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.003s
I20260806 03:42:31.246963 843 raft_consensus.cc:348] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.247879 843 raft_consensus.cc:374] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.248212 843 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.249114 843 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.252074 843 ts_tablet_manager.cc:1428] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:42:31.255489 845 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.255878 843 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.255967 845 ts_tablet_manager.cc:1397] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.015s
I20260806 03:42:31.259611 845 raft_consensus.cc:348] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.260313 845 raft_consensus.cc:374] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.260574 845 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.261252 845 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.263521 843 tablet_bootstrap.cc:654] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.270855 1315 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.271456 1315 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.272810 845 ts_tablet_manager.cc:1428] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260806 03:42:31.272512 402 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:31.273651 1315 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.273834 845 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.278936 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.279520 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.279627 668 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:31.281364 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:42:31.282687 727 leader_election.cc:341] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.283213 842 tablet_bootstrap.cc:654] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.283378 1236 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.285079 1236 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.285473 1236 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.284353 1315 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.292011 1315 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.293937 1315 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.296428 845 tablet_bootstrap.cc:654] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.298023 1236 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.299978 1236 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.306919 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.312947 843 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.313598 843 ts_tablet_manager.cc:1397] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.018s
I20260806 03:42:31.316313 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.319979 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.320981 668 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.324843 843 raft_consensus.cc:348] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.324868 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:31.330541 843 raft_consensus.cc:374] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.330981 843 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.331856 843 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:31.342077 727 leader_election.cc:341] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:31.343144 727 leader_election.cc:341] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:31.344429 727 leader_election.cc:341] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.344893 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.345871 1236 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.355648 843 ts_tablet_manager.cc:1428] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.042s user 0.006s sys 0.005s
I20260806 03:42:31.357437 843 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.369163 843 tablet_bootstrap.cc:654] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.387950 668 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:31.389063 843 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.389778 843 ts_tablet_manager.cc:1397] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.003s
I20260806 03:42:31.390441 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:31.391232 1236 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.391733 1236 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:31.392377 843 raft_consensus.cc:348] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.393107 843 raft_consensus.cc:374] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.393193 842 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:31.392848 1236 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.389180 845 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.393658 842 ts_tablet_manager.cc:1397] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.154s user 0.020s sys 0.045s
I20260806 03:42:31.393922 845 ts_tablet_manager.cc:1397] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.120s user 0.023s sys 0.063s
I20260806 03:42:31.393388 843 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.396472 843 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.396768 845 raft_consensus.cc:348] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.397512 845 raft_consensus.cc:374] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.397779 845 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.398664 843 ts_tablet_manager.cc:1428] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:42:31.398422 845 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.399646 843 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.399971 842 raft_consensus.cc:348] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.400651 842 raft_consensus.cc:374] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.400919 842 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.401087 1253 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:31.401567 842 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.403064 842 ts_tablet_manager.cc:1428] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 03:42:31.405252 845 ts_tablet_manager.cc:1428] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260806 03:42:31.406302 845 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.415385 843 tablet_bootstrap.cc:654] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.420975 842 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:31.429536 845 tablet_bootstrap.cc:654] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.438941 1187 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.441340 1187 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:31.441879 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:31.443280 1187 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.444289 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.444671 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.446250 616 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:31.447826 842 tablet_bootstrap.cc:654] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.447485 1187 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.452080 843 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.452554 843 ts_tablet_manager.cc:1397] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.009s
I20260806 03:42:31.458633 402 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:31.460645 1110 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:31.464649 1110 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:31.470139 843 raft_consensus.cc:348] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.471127 843 raft_consensus.cc:374] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.471470 843 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.478937 843 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.485446 843 ts_tablet_manager.cc:1428] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260806 03:42:31.493888 842 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:31.494539 842 ts_tablet_manager.cc:1397] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.074s user 0.008s sys 0.020s
W20260806 03:42:31.497969 498 consensus_peers.cc:487] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:31.499715 843 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.516924 1236 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.517447 1236 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.519438 1236 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.524056 845 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.524519 845 ts_tablet_manager.cc:1397] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.118s user 0.029s sys 0.005s
I20260806 03:42:31.525548 842 raft_consensus.cc:348] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.526674 842 raft_consensus.cc:374] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.527185 842 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.527657 902 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.527561 845 raft_consensus.cc:348] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.528383 845 raft_consensus.cc:374] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.528167 902 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.528697 845 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.528503 842 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.529278 845 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.529937 902 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.531339 845 ts_tablet_manager.cc:1428] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:42:31.532019 843 tablet_bootstrap.cc:654] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.531970 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.532351 1356 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:42:31.533300 727 leader_election.cc:341] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.532799 1356 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.537518 1356 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.539336 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.540053 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.539975 775 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:31.541662 845 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
W20260806 03:42:31.541730 727 leader_election.cc:341] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.543300 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:31.545738 1356 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.546104 1356 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.556278 842 ts_tablet_manager.cc:1428] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260806 03:42:31.558995 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.561000 842 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:31.558475 1356 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.569936 1236 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.570519 1236 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.571624 845 tablet_bootstrap.cc:654] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.573163 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.573783 562 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:31.576445 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.577514 1236 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.578125 661 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.579097 843 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
W20260806 03:42:31.579608 614 leader_election.cc:341] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.579620 843 ts_tablet_manager.cc:1397] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.082s user 0.015s sys 0.010s
I20260806 03:42:31.580953 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:31.582782 727 leader_election.cc:341] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.586786 843 raft_consensus.cc:348] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.588603 843 raft_consensus.cc:374] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.588861 843 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.589481 843 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.590603 842 tablet_bootstrap.cc:654] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.594727 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.601547 843 ts_tablet_manager.cc:1428] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20260806 03:42:31.602664 843 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.603899 1253 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:31.607349 845 tablet_bootstrap.cc:492] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.607816 845 ts_tablet_manager.cc:1397] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.005s
I20260806 03:42:31.608181 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.608925 902 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.608158 843 tablet_bootstrap.cc:654] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.609252 902 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.609563 902 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.610620 845 raft_consensus.cc:348] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.611286 845 raft_consensus.cc:374] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.611549 845 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
W20260806 03:42:31.611982 727 leader_election.cc:341] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.612418 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.613404 1236 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.612111 845 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.617024 1356 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.619648 845 ts_tablet_manager.cc:1428] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 03:42:31.620587 845 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.623668 902 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:31.624209 727 leader_election.cc:341] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.653784 727 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.654736 1256 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.632589 1253 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:31.657873 902 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.640554 1236 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.643915 1348 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.644055 845 tablet_bootstrap.cc:654] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.661005 1348 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.663280 1348 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.660965 1236 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.664690 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.665385 1236 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.669111 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.670455 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.639545 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:31.672649 1348 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:42:31.672931 727 leader_election.cc:341] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.673170 1348 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.652868 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:31.673811 661 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.674098 775 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.675187 1348 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:31.673600 727 leader_election.cc:341] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.681020 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:31.692507 614 leader_election.cc:341] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:31.693264 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 50398us
I20260806 03:42:31.694814 1356 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.700356 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.701433 843 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.701894 843 ts_tablet_manager.cc:1397] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.099s user 0.022s sys 0.020s
I20260806 03:42:31.703868 843 raft_consensus.cc:348] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.704375 843 raft_consensus.cc:374] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.704615 843 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.705202 843 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.707007 843 ts_tablet_manager.cc:1428] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260806 03:42:31.707739 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.708587 558 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:42:31.709411 843 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Bootstrap starting.
I20260806 03:42:31.710229 842 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:31.711885 842 ts_tablet_manager.cc:1397] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.151s user 0.017s sys 0.015s
I20260806 03:42:31.714877 842 raft_consensus.cc:348] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.715528 843 tablet_bootstrap.cc:654] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.715595 842 raft_consensus.cc:374] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.715973 842 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.716579 842 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.718096 842 ts_tablet_manager.cc:1428] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 03:42:31.736408 843 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: No bootstrap required, opened a new log
I20260806 03:42:31.736932 843 ts_tablet_manager.cc:1397] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.001s
I20260806 03:42:31.737250 845 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.737704 845 ts_tablet_manager.cc:1397] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.117s user 0.011s sys 0.008s
I20260806 03:42:31.740249 845 raft_consensus.cc:348] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.741297 1353 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.741638 845 raft_consensus.cc:374] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.741958 845 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.741772 1353 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.742970 845 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.744005 1353 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.744441 842 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:31.744618 1353 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.745332 845 ts_tablet_manager.cc:1428] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260806 03:42:31.745124 1353 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.743470 843 raft_consensus.cc:348] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.746405 845 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.748032 843 raft_consensus.cc:374] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.748391 843 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.749140 843 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.750672 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:31.751196 562 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.747203 1353 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.706001 775 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:31.757139 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:31.772063 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:31.773422 614 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:31.774828 1187 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.776060 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.777967 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:31.780158 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:31.778950 843 ts_tablet_manager.cc:1428] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent starting tablet: real 0.042s user 0.005s sys 0.002s
I20260806 03:42:31.784358 562 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:31.786813 1348 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.802582 902 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:42:31.803169 727 leader_election.cc:341] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.804596 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.810734 845 tablet_bootstrap.cc:654] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.817025 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.817732 771 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:31.821671 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:31.822190 498 leader_election.cc:341] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.806229 1256 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.822573 770 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:31.804903 1348 raft_consensus.cc:686] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:42:31.824771 498 leader_election.cc:341] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:31.828132 842 tablet_bootstrap.cc:654] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:31.829912 1432 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.830425 1187 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:42:31.828953 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61763.5us
I20260806 03:42:31.830509 1432 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.835325 1348 consensus_queue.cc:237] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.838874 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.839885 902 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:31.842808 902 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:31.843850 902 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.878096 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.879317 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:31.869191 1356 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:31.884428 1110 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.885830 1406 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:31.895241 1406 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.896924 1406 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.898149 1433 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.894719 1110 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:31.898814 1110 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.898782 1433 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.900872 845 tablet_bootstrap.cc:492] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:31.901340 845 ts_tablet_manager.cc:1397] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.155s user 0.010s sys 0.007s
I20260806 03:42:31.912523 1110 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.912557 1406 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.917028 845 raft_consensus.cc:348] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.917769 845 raft_consensus.cc:374] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:31.918027 845 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:31.918843 1110 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.900476 1432 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.922986 845 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:31.939525 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 60280.9us
I20260806 03:42:31.950752 845 ts_tablet_manager.cc:1428] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s
I20260806 03:42:31.953150 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.953797 661 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.929103 1406 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:31.954473 1439 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:31.959985 1439 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:31.957850 1433 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:31.962447 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:31.962563 1433 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:31.964393 563 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.939121 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:31.965366 769 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:31.964125 1441 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260806 03:42:31.992497 845 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:31.966914 402 catalog_manager.cc:5495] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:32.001691 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:32.002745 775 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:31.962203 1439 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.017727 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:32.020196 842 tablet_bootstrap.cc:492] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:32.020766 842 ts_tablet_manager.cc:1397] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.277s user 0.027s sys 0.015s
W20260806 03:42:32.021800 498 leader_election.cc:341] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.031091 775 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:32.034092 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:32.034845 772 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.041203 1456 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:32.038247 403 catalog_manager.cc:5495] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:32.062724 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:32.064844 1430 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.065346 1430 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:32.065270 842 raft_consensus.cc:348] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.066139 842 raft_consensus.cc:374] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:32.066309 1430 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.066609 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:32.068246 842 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:32.069067 842 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.071926 845 tablet_bootstrap.cc:654] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:32.075301 772 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:32.081068 842 ts_tablet_manager.cc:1428] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.060s user 0.007s sys 0.001s
I20260806 03:42:32.052040 1456 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.082506 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:32.083606 1456 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.087886 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:32.088639 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:32.089517 1436 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:32.089872 1436 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:32.090180 1436 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:42:32.090554 498 leader_election.cc:341] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.092062 842 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:32.095280 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:32.095984 770 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:32.098709 661 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:32.101540 498 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:42:32.103637 727 leader_election.cc:341] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.107424 1456 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.108127 1456 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:32.108997 1456 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.114638 616 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:42:32.119071 727 leader_election.cc:341] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.119554 727 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:32.120436 1442 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:32.121455 845 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:32.121934 845 ts_tablet_manager.cc:1397] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.130s user 0.016s sys 0.005s
W20260806 03:42:32.123279 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 59382.7us
I20260806 03:42:32.124872 845 raft_consensus.cc:348] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.125885 845 raft_consensus.cc:374] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:32.126987 845 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:32.118335 1439 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:32.128711 1439 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:32.129083 1439 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.128785 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:32.127851 845 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.135339 1444 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:32.145411 845 ts_tablet_manager.cc:1428] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s
I20260806 03:42:32.148171 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:32.148823 772 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:32.116679 1436 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:32.148342 614 leader_election.cc:341] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.152196 1436 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:32.155565 614 leader_election.cc:341] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.157830 845 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:32.137266 1444 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.175882 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:32.183987 501 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:32.185322 1403 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:32.185891 1403 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:32.186288 1403 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.165797 1236 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20260806 03:42:32.259337 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:32.266605 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 81240.3us
I20260806 03:42:32.267354 845 tablet_bootstrap.cc:654] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
W20260806 03:42:32.274155 614 leader_election.cc:341] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:32.278738 1457 log.cc:924] Time spent T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260806 03:42:32.282640 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:32.270757 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:32.268400 1444 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.303938 561 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.305908 842 tablet_bootstrap.cc:654] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:32.299295 769 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.335139 501 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:42:32.345654 1469 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260806 03:42:32.353495 1439 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.360833 1439 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.346812 1110 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:42:32.378925 1475 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
W20260806 03:42:32.423728 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 88661us
I20260806 03:42:32.488237 1403 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.495589 842 tablet_bootstrap.cc:492] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:32.496088 842 ts_tablet_manager.cc:1397] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.404s user 0.016s sys 0.003s
I20260806 03:42:32.520166 1403 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.491464 1432 consensus_queue.cc:1035] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260806 03:42:32.541541 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:32.544687 1471 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:32.545290 1471 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.552279 1456 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:32.559026 1456 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.572903 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:32.576471 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:32.579510 1456 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.580294 1471 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:42:32.590646 1492 log.cc:924] Time spent T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 03:42:32.596726 842 raft_consensus.cc:348] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.597821 842 raft_consensus.cc:374] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:32.598706 403 catalog_manager.cc:5495] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:32.599783 614 leader_election.cc:341] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:32.598183 842 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:32.599695 1471 consensus_queue.cc:1035] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:32.609278 403 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:32.614634 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:32.613572 842 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.624785 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:32.625563 771 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.629787 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:32.630456 770 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.633612 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:32.661659 769 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:32.661903 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:42:32.674996 498 leader_election.cc:341] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:32.683743 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:32.686560 842 ts_tablet_manager.cc:1428] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.188s user 0.008s sys 0.002s
W20260806 03:42:32.688105 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 151165us
I20260806 03:42:32.691092 1416 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:32.695385 842 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:32.722555 1253 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:32.724466 501 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:32.763590 845 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:32.764278 845 ts_tablet_manager.cc:1397] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.607s user 0.031s sys 0.014s
I20260806 03:42:32.815860 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:32.829314 845 raft_consensus.cc:348] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.832638 845 raft_consensus.cc:374] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:32.835533 845 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:32.836546 845 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.844627 842 tablet_bootstrap.cc:654] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:32.814107 1406 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:42:32.852640 771 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:42:32.822913 1432 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:32.847936 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:32.854728 1432 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.855214 1432 raft_consensus.cc:686] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:32.856037 1432 consensus_queue.cc:237] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:32.860437 498 consensus_peers.cc:487] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:32.864135 845 ts_tablet_manager.cc:1428] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.099s user 0.008s sys 0.002s
I20260806 03:42:32.864461 1491 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
W20260806 03:42:32.882874 727 consensus_peers.cc:487] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:32.865394 770 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:32.886497 501 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:32.881585 1485 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260806 03:42:32.894134 1491 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.903676 1491 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:32.904577 1491 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.910245 616 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:32.911968 1439 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:32.912472 1439 raft_consensus.cc:686] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:32.913419 1439 consensus_queue.cc:237] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.882618 1353 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:32.918208 1353 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:32.922350 1353 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:32.925582 671 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:32.941705 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:32.942941 773 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 1: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:42:33.001873 845 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.024582 845 tablet_bootstrap.cc:654] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.037986 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.043247 845 tablet_bootstrap.cc:492] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.043933 845 ts_tablet_manager.cc:1397] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.002s
I20260806 03:42:33.049449 845 raft_consensus.cc:348] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.050514 845 raft_consensus.cc:374] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.050858 845 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.052138 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.053855 1491 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.054308 842 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.054929 842 ts_tablet_manager.cc:1397] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.360s user 0.011s sys 0.013s
I20260806 03:42:33.057427 842 raft_consensus.cc:348] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.058133 842 raft_consensus.cc:374] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.058403 842 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.059300 842 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.061894 842 ts_tablet_manager.cc:1428] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:42:33.063225 842 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:33.064252 1491 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.069398 1187 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.069970 1187 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.069617 845 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.072696 845 ts_tablet_manager.cc:1428] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.028s user 0.005s sys 0.004s
I20260806 03:42:33.073433 1187 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.073684 845 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.078644 842 tablet_bootstrap.cc:654] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.078751 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.079727 845 tablet_bootstrap.cc:654] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.108664 664 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.111179 402 catalog_manager.cc:5495] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:33.116864 845 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.117364 845 ts_tablet_manager.cc:1397] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.008s
I20260806 03:42:33.119693 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:33.120024 845 raft_consensus.cc:348] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.120771 845 raft_consensus.cc:374] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.121181 845 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.121915 845 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.125221 614 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:33.127734 845 ts_tablet_manager.cc:1428] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 03:42:33.128786 845 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.143057 842 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.146291 842 ts_tablet_manager.cc:1397] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.003s
I20260806 03:42:33.145804 845 tablet_bootstrap.cc:654] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.149405 842 raft_consensus.cc:348] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.155325 842 raft_consensus.cc:374] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.155653 842 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.155102 402 catalog_manager.cc:5495] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:33.156471 842 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.157805 1507 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.158288 1507 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.160710 1507 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:33.162077 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.159873 842 ts_tablet_manager.cc:1428] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 03:42:33.163583 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:33.163836 727 leader_election.cc:341] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:33.164830 727 leader_election.cc:341] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:33.165238 842 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:33.165234 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.157025 402 catalog_manager.cc:5495] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:33.167035 1507 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.177918 1187 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.178648 845 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.179208 845 ts_tablet_manager.cc:1397] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.010s
I20260806 03:42:33.188784 842 tablet_bootstrap.cc:654] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.189700 845 raft_consensus.cc:348] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.193753 845 raft_consensus.cc:374] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.194115 845 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.195024 845 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.201069 845 ts_tablet_manager.cc:1428] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260806 03:42:33.201983 845 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.208479 845 tablet_bootstrap.cc:654] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.208696 842 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.209139 842 ts_tablet_manager.cc:1397] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.001s
I20260806 03:42:33.211767 842 raft_consensus.cc:348] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.212698 842 raft_consensus.cc:374] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.213032 842 raft_consensus.cc:729] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.213941 842 consensus_queue.cc:260] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.218055 842 ts_tablet_manager.cc:1428] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:42:33.219125 842 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:33.224853 842 tablet_bootstrap.cc:654] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.244156 842 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.244627 842 ts_tablet_manager.cc:1397] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.008s
W20260806 03:42:33.245199 614 consensus_peers.cc:487] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 -> Peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Couldn't send request to peer ada5ef6656cb4420baf77a4b4342909e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:33.247056 842 raft_consensus.cc:348] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.247792 842 raft_consensus.cc:374] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.248039 842 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.248672 842 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.251261 842 ts_tablet_manager.cc:1428] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260806 03:42:33.252228 842 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:33.256429 845 tablet_bootstrap.cc:492] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.256662 1187 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:33.257011 845 ts_tablet_manager.cc:1397] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.008s
I20260806 03:42:33.257867 842 tablet_bootstrap.cc:654] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.259855 845 raft_consensus.cc:348] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.260671 845 raft_consensus.cc:374] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.260957 845 raft_consensus.cc:729] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.261622 845 consensus_queue.cc:260] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.270824 845 ts_tablet_manager.cc:1428] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260806 03:42:33.271854 845 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.276160 1315 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.276840 1315 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.279241 842 tablet_bootstrap.cc:492] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.279752 1315 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:33.279770 842 ts_tablet_manager.cc:1397] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.009s
I20260806 03:42:33.281404 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:33.281561 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.282112 561 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 0.
I20260806 03:42:33.282521 845 tablet_bootstrap.cc:654] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.282639 842 raft_consensus.cc:348] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:33.282963 727 leader_election.cc:341] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:33.283731 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.284760 1315 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.285094 1315 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.285434 1315 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.291905 842 raft_consensus.cc:374] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.292232 842 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.293443 842 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.294055 845 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.294611 845 ts_tablet_manager.cc:1397] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.004s
I20260806 03:42:33.296260 842 ts_tablet_manager.cc:1428] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260806 03:42:33.297291 842 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Bootstrap starting.
I20260806 03:42:33.297016 845 raft_consensus.cc:348] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.298645 845 raft_consensus.cc:374] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.298974 845 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.299878 845 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.303221 842 tablet_bootstrap.cc:654] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.304998 845 ts_tablet_manager.cc:1428] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260806 03:42:33.307103 1315 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.309055 1315 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:33.310681 845 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.313808 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:33.314464 546 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.315296 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:33.319759 1403 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:42:33.320483 727 leader_election.cc:341] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:33.324455 1485 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.325140 1485 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.320401 1403 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.331167 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.331844 664 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.332093 1485 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.333843 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.334510 775 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.335651 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.336022 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:33.338073 1485 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.338668 1485 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.339203 1485 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.347242 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.349671 498 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.350620 1353 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.353924 1353 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.354423 1353 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.356410 1353 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.360884 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.361847 768 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.363773 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.364482 664 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.333778 1403 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.370178 501 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:33.324476 546 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:42:33.373045 1353 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.374089 1353 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.374687 1353 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.377418 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.378332 1507 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:33.378844 1507 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:33.379707 1507 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.386523 1479 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.387168 1479 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.392803 1479 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:33.396926 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.397843 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:42:33.416975 727 leader_election.cc:341] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:42:33.418965 727 leader_election.cc:341] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:33.419471 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.422097 845 tablet_bootstrap.cc:654] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.422147 902 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.416697 1439 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.437165 1432 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260806 03:42:33.437382 1439 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.446419 1187 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.466643 1491 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.469375 1491 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.470278 1439 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.471513 1491 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.474516 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:42:33.455453 1187 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.480974 663 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.481158 1432 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.484176 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:33.486096 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.486575 1187 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.495924 768 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.497890 501 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:33.487293 1485 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.488859 842 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: No bootstrap required, opened a new log
I20260806 03:42:33.481875 1432 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.493563 401 catalog_manager.cc:5495] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:33.501291 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.502123 842 ts_tablet_manager.cc:1397] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Time spent bootstrapping tablet: real 0.205s user 0.040s sys 0.065s
I20260806 03:42:33.500900 1491 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.504657 775 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 0.
I20260806 03:42:33.506894 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:33.506240 1491 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.514359 1491 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.517005 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:33.511549 1439 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.510277 663 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.517707 1439 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.518640 1439 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.515544 1432 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.505309 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:33.505769 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.526935 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.528688 1444 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:33.530130 671 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.531819 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:33.538471 768 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:33.539675 616 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:33.540747 1545 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:33.517685 1353 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.526170 842 raft_consensus.cc:348] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.546574 842 raft_consensus.cc:374] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.548264 842 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.549212 842 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.547304 1444 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 03:42:33.551641 1545 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.559170 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:33.559998 671 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.562719 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:33.563406 768 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.565174 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:33.565807 664 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.589020 664 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:33.590019 842 ts_tablet_manager.cc:1428] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Time spent starting tablet: real 0.085s user 0.010s sys 0.002s
I20260806 03:42:33.590221 498 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:33.596012 1403 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:33.597035 1403 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:33.598093 1403 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.569003 1353 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.603257 845 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.603958 845 ts_tablet_manager.cc:1397] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.293s user 0.013s sys 0.004s
I20260806 03:42:33.603257 671 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:33.605552 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:33.609393 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:33.610219 775 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.620206 775 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:33.609447 1485 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.634284 1491 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.635620 546 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:33.637976 1491 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.608908 1353 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:33.639549 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:42:33.631531 1550 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260806 03:42:33.644126 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:33.644758 770 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.645118 1439 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.643610 768 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:33.649384 671 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 1: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:42:33.656076 1444 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.660096 1444 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.660844 845 raft_consensus.cc:348] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.661651 845 raft_consensus.cc:374] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.661952 845 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:33.662614 845 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.663165 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:42:33.664810 498 leader_election.cc:341] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:42:33.666230 1444 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.664844 845 ts_tablet_manager.cc:1428] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20260806 03:42:33.668774 845 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:33.672613 1439 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.674002 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:33.674892 546 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 1: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:42:33.678951 1432 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.679744 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:33.681903 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:33.683410 1432 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:42:33.687731 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:33.688450 776 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 0.
I20260806 03:42:33.688649 845 tablet_bootstrap.cc:654] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:33.691968 501 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.692754 1253 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:42:33.693192 1253 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:42:33.693528 1253 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.703946 770 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:33.850843 1353 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:33.852355 1439 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:42:33.852859 1439 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.851864 1353 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.864568 547 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:33.886025 845 tablet_bootstrap.cc:492] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:33.886698 845 ts_tablet_manager.cc:1397] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.218s user 0.033s sys 0.016s
I20260806 03:42:33.923331 1253 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.927974 1253 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.925812 1439 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:33.928282 845 raft_consensus.cc:348] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:33.929163 845 raft_consensus.cc:374] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:33.929407 845 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
W20260806 03:42:33.938283 472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:42:33.930045 845 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:33.946655 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56607.6us
I20260806 03:42:33.948315 845 ts_tablet_manager.cc:1428] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.061s user 0.007s sys 0.000s
I20260806 03:42:33.948518 501 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:33.977104 1556 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:34.003450 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:42:34.012463 1556 raft_consensus.cc:686] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:34.013430 1556 consensus_queue.cc:237] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:34.047221 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:34.048282 659 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.047226 667 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.058350 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:42:34.060343 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:34.060973 767 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.076786 767 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:34.080194 501 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:34.081094 1564 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:42:34.081799 1564 raft_consensus.cc:686] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:34.082767 1564 consensus_queue.cc:237] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:42:34.096975 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 121357us
W20260806 03:42:34.103616 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53008.1us
W20260806 03:42:34.123544 727 consensus_peers.cc:487] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Couldn't send request to peer d8469561a52c4bbc81dce786f1211ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:34.134191 614 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:42:34.137868 659 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:42:34.145071 845 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Bootstrap starting.
I20260806 03:42:34.148712 402 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:34.152882 402 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:34.182214 845 tablet_bootstrap.cc:654] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Neither blocks nor log segments found. Creating new log.
I20260806 03:42:34.183671 1566 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.184309 1439 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:42:34.186616 902 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.229456 845 tablet_bootstrap.cc:492] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: No bootstrap required, opened a new log
I20260806 03:42:34.229975 845 ts_tablet_manager.cc:1397] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.002s
I20260806 03:42:34.232836 845 raft_consensus.cc:348] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:34.233583 845 raft_consensus.cc:374] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:42:34.233847 845 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Initialized, Role: FOLLOWER
I20260806 03:42:34.234567 845 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:34.236157 845 ts_tablet_manager.cc:1428] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 03:42:34.250290 1564 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:34.293416 402 catalog_manager.cc:5495] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:34.295706 402 catalog_manager.cc:5495] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 0 to 1, leader changed from <none> to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:34.335371 1564 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.346063 1566 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.408419 664 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:42:34.423929 1353 consensus_queue.cc:1035] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:34.438786 1353 consensus_queue.cc:1035] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.629773 1428 consensus_queue.cc:1035] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:42:34.645813 1428 consensus_queue.cc:1035] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:42:35.080402 31273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:42:35.103717 1608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:35.131525 1609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:42:35.138988 1614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:42:35.151593 31273 server_base.cc:1034] running on GCE node
I20260806 03:42:35.154167 31273 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:42:35.154547 31273 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:42:35.154762 31273 hybrid_clock.cc:648] HybridClock initialized: now 1785987755154745 us; error 0 us; skew 500 ppm
I20260806 03:42:35.155460 31273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:42:35.164870 31273 webserver.cc:460] Webserver started at http://127.30.138.68:37669/ using document root <none> and password file <none>
I20260806 03:42:35.165400 31273 fs_manager.cc:359] Metadata directory not provided
I20260806 03:42:35.165645 31273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:42:35.165937 31273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:42:35.167342 31273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-3-root/instance:
uuid: "1914b652c9fb49b694d71a5f5f5752ee"
format_stamp: "Formatted at 2026-08-06 03:42:35 on dist-test-slave-tk9z"
I20260806 03:42:35.172963 31273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 03:42:35.180007 1621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:35.182315 31273 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s
I20260806 03:42:35.182842 31273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-3-root
uuid: "1914b652c9fb49b694d71a5f5f5752ee"
format_stamp: "Formatted at 2026-08-06 03:42:35 on dist-test-slave-tk9z"
I20260806 03:42:35.183403 31273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:42:35.203944 31273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:42:35.206079 31273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:42:35.207610 31273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:42:35.210060 31273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:42:35.210294 31273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:35.210664 31273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:42:35.210842 31273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:42:35.367376 31273 rpc_server.cc:307] RPC server started. Bound to: 127.30.138.68:33823
I20260806 03:42:35.367602 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.138.68:33823 every 8 connection(s)
I20260806 03:42:35.380285 31273 auto_leader_rebalancer-test.cc:413] add a tserver: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:35.416126 1714 heartbeater.cc:347] Connected to a master server at 127.30.138.126:46149
I20260806 03:42:35.416651 1714 heartbeater.cc:464] Registering TS with master...
I20260806 03:42:35.417667 1714 heartbeater.cc:511] Master 127.30.138.126:46149 requested a full tablet report, sending...
I20260806 03:42:35.421180 403 ts_manager.cc:194] Registered new tserver with Master: 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:42:35.423467 403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51378
I20260806 03:42:36.071964 768 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:36.080560 667 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.081984 1507 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:36.082238 546 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.090325 1507 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:36.108673 1507 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.109908 667 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.125785 546 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:36.143587 728 consensus_peers.cc:487] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e1fbe5a0e51744568ce535d54093a890. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:36.147285 403 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:36.170256 546 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:36.199254 768 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.200273 667 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.200843 1615 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:36.202292 1428 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:36.215819 1353 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:36.223255 499 consensus_peers.cc:487] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 921ac79ba4d94c039b073809e4a17f11. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:36.227296 667 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.244845 775 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:36.257596 403 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.268220 768 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:36.291664 728 consensus_peers.cc:487] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b19e6696d8d549f58265db62b449e706. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:36.292838 667 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.295781 546 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.295619 1569 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:36.300891 1507 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:36.308727 1569 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.310388 659 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.313846 546 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.333355 403 catalog_manager.cc:5495] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:36.361634 659 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:36.370903 775 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.373526 1432 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:36.380061 547 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:36.382184 1603 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:36.397118 1432 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.400750 775 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:36.402266 615 consensus_peers.cc:487] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44d06aad0812478bb2270a2c80917306. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:36.411331 547 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.428869 1714 heartbeater.cc:503] Master 127.30.138.126:46149 was elected leader, sending a full tablet report...
I20260806 03:42:36.433295 402 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:36.663640 1771 ts_tablet_manager.cc:927] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:36.668524 1771 tablet_copy_client.cc:316] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:37.246126 1795 ts_tablet_manager.cc:927] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:37.248999 1794 ts_tablet_manager.cc:927] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:37.278955 1794 tablet_copy_client.cc:316] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:37.333743 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet e1fbe5a0e51744568ce535d54093a890 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:37.334672 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet e1fbe5a0e51744568ce535d54093a890 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-e1fbe5a0e51744568ce535d54093a890
I20260806 03:42:37.338692 789 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet b19e6696d8d549f58265db62b449e706 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:37.339507 789 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet b19e6696d8d549f58265db62b449e706 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-b19e6696d8d549f58265db62b449e706
I20260806 03:42:37.360029 789 tablet_copy_source_session.cc:205] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:37.363603 790 tablet_copy_source_session.cc:205] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:37.367460 1771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1fbe5a0e51744568ce535d54093a890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.368490 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b19e6696d8d549f58265db62b449e706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.375351 1795 tablet_copy_client.cc:316] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
W20260806 03:42:37.436482 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 66577.9us
I20260806 03:42:37.483570 1794 tablet_copy_client.cc:791] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:37.484818 1794 tablet_copy_client.cc:655] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:37.494017 1794 tablet_copy_client.cc:531] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:37.515182 1802 ts_tablet_manager.cc:927] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:37.525398 1802 tablet_copy_client.cc:316] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:42:37.543617 1771 tablet_copy_client.cc:791] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:37.544342 1771 tablet_copy_client.cc:655] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:37.557245 1794 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:37.558480 1771 tablet_copy_client.cc:531] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:37.623383 1794 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:37.625422 1794 tablet_bootstrap.cc:492] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:37.627342 1794 ts_tablet_manager.cc:1397] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.070s user 0.026s sys 0.020s
I20260806 03:42:37.640115 1794 raft_consensus.cc:348] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:37.641947 1794 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:37.643241 1794 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:37.644935 1771 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:37.715031 1794 ts_tablet_manager.cc:1428] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.087s user 0.022s sys 0.026s
I20260806 03:42:37.718891 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-b19e6696d8d549f58265db62b449e706 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:37.719702 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-b19e6696d8d549f58265db62b449e706 on tablet b19e6696d8d549f58265db62b449e706 with peer 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:42:37.721555 805 debug-util.cc:398] Leaking SignalData structure 0x7b0800111f60 after lost signal to thread 31279
W20260806 03:42:37.723549 805 debug-util.cc:398] Leaking SignalData structure 0x7b0800200300 after lost signal to thread 450
I20260806 03:42:37.724633 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 921ac79ba4d94c039b073809e4a17f11 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:37.725672 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 921ac79ba4d94c039b073809e4a17f11 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-921ac79ba4d94c039b073809e4a17f11
W20260806 03:42:37.730875 805 debug-util.cc:398] Leaking SignalData structure 0x7b080030b2c0 after lost signal to thread 811
W20260806 03:42:37.732628 805 debug-util.cc:398] Leaking SignalData structure 0x7b080030b0e0 after lost signal to thread 1712
W20260806 03:42:37.749717 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45518.1us
I20260806 03:42:37.763185 575 tablet_copy_source_session.cc:205] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:37.769608 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921ac79ba4d94c039b073809e4a17f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.805357 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet 44d06aad0812478bb2270a2c80917306 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:42:37.806645 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet 44d06aad0812478bb2270a2c80917306 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-44d06aad0812478bb2270a2c80917306
I20260806 03:42:37.855502 691 tablet_copy_source_session.cc:205] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:37.862460 1802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44d06aad0812478bb2270a2c80917306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:37.880754 1795 tablet_copy_client.cc:791] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:37.899016 1795 tablet_copy_client.cc:655] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:37.900058 1802 tablet_copy_client.cc:791] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:37.900911 1802 tablet_copy_client.cc:655] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:37.907507 1795 tablet_copy_client.cc:531] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:37.919085 1802 tablet_copy_client.cc:531] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:37.946715 1771 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:37.947990 1771 tablet_bootstrap.cc:492] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:37.950971 1802 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:37.951864 1771 ts_tablet_manager.cc:1397] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.307s user 0.025s sys 0.032s
I20260806 03:42:37.960608 1795 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:37.973975 1771 raft_consensus.cc:348] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:37.979872 1771 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:37.980891 1771 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:37.989719 1771 ts_tablet_manager.cc:1428] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.031s user 0.009s sys 0.000s
I20260806 03:42:37.992894 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-e1fbe5a0e51744568ce535d54093a890 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:37.993708 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-e1fbe5a0e51744568ce535d54093a890 on tablet e1fbe5a0e51744568ce535d54093a890 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:38.042670 1802 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:38.044291 1802 tablet_bootstrap.cc:492] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:38.045197 1802 ts_tablet_manager.cc:1397] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.095s user 0.019s sys 0.040s
I20260806 03:42:38.051119 1802 raft_consensus.cc:348] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:38.052829 1802 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:38.053913 1802 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:38.059226 1802 ts_tablet_manager.cc:1428] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260806 03:42:38.061823 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-44d06aad0812478bb2270a2c80917306 received from {username='slave'} at 127.0.0.1:46602
I20260806 03:42:38.062263 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-44d06aad0812478bb2270a2c80917306 on tablet 44d06aad0812478bb2270a2c80917306 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:38.074122 1670 raft_consensus.cc:1203] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:38.087386 1670 raft_consensus.cc:1203] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:38.189334 1670 raft_consensus.cc:1203] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:38.225574 1795 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:38.226946 1795 tablet_bootstrap.cc:492] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:38.228107 1795 ts_tablet_manager.cc:1397] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.268s user 0.031s sys 0.032s
I20260806 03:42:38.234457 1795 raft_consensus.cc:348] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:38.236914 1795 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:38.238250 1795 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:38.259214 1795 ts_tablet_manager.cc:1428] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.031s user 0.008s sys 0.008s
I20260806 03:42:38.261241 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-921ac79ba4d94c039b073809e4a17f11 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:38.261861 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-921ac79ba4d94c039b073809e4a17f11 on tablet 921ac79ba4d94c039b073809e4a17f11 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:38.537572 1670 raft_consensus.cc:1203] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:38.612911 1746 raft_consensus.cc:1050] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:38.617446 1746 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:38.639709 1569 raft_consensus.cc:1050] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:38.649353 1569 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:38.716219 1670 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.724282 1791 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.753788 659 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.756327 546 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.758816 1818 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:42:38.762799 1807 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:42:38.785137 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 52695us
I20260806 03:42:38.805002 1784 raft_consensus.cc:1050] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:38.817014 1784 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:38.827335 663 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.827275 1670 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.829612 1807 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.833490 1569 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.847070 1791 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:38.851372 564 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.850983 546 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:38.853188 1807 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.874563 668 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:38.910980 1671 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.924322 771 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:38.923636 402 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:38.929198 1821 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.937552 1810 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:38.954792 1746 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:38.960323 564 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.001017 546 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:39.003633 1810 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:42:39.019057 1670 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.027752 1671 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.025633 1830 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.044559 1668 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.077867 771 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.080057 546 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.102809 1744 raft_consensus.cc:1050] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:39.111222 668 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.117143 1744 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:39.163095 768 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:39.165723 1800 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:39.168915 668 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:39.166036 1668 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:39.183992 1428 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:39.190745 1748 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:42:39.211571 1840 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260806 03:42:39.249712 775 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:39.257197 403 catalog_manager.cc:5495] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.261693 1670 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.274520 1800 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.276657 1671 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.293514 1746 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.279346 667 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.297557 667 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.306237 1791 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.306106 771 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.289677 403 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.308980 1818 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.320508 1670 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.339236 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e1fbe5a0e51744568ce535d54093a890 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:39.349006 767 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:39.355919 668 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:39.353753 667 raft_consensus.cc:2932] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.367045 767 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.368790 1668 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.368956 1821 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260806 03:42:39.378857 1791 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.367712 1671 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.389070 1784 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.392580 546 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.398759 1830 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:42:39.413136 378 catalog_manager.cc:5018] ChangeConfig:REMOVE_PEER RPC for tablet e1fbe5a0e51744568ce535d54093a890 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20260806 03:42:39.413455 1746 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.416236 775 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.423125 1671 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.426898 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 44d06aad0812478bb2270a2c80917306 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:39.425472 1668 raft_consensus.cc:2932] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.433336 1818 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.441164 403 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.453801 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b19e6696d8d549f58265db62b449e706 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:39.465997 546 raft_consensus.cc:2932] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.507378 564 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:39.518508 1668 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.521263 1744 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.522037 659 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:39.505681 401 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:39.525650 1779 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:39.532160 1748 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.542542 1671 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.552834 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 921ac79ba4d94c039b073809e4a17f11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:39.572996 400 catalog_manager.cc:5495] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.553252 667 raft_consensus.cc:2932] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.574957 400 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.583845 536 tablet_service.cc:1514] Processing DeleteTablet for tablet e1fbe5a0e51744568ce535d54093a890 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:39.600229 1849 tablet_replica.cc:330] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:39.607378 1849 raft_consensus.cc:2229] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:39.608306 1849 raft_consensus.cc:2258] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:39.616050 536 tablet_service.cc:1514] Processing DeleteTablet for tablet 44d06aad0812478bb2270a2c80917306 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:39.620024 648 tablet_service.cc:1514] Processing DeleteTablet for tablet b19e6696d8d549f58265db62b449e706 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:39.616653 401 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:39.622097 1849 ts_tablet_manager.cc:1905] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:39.624706 1851 tablet_replica.cc:330] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:39.625847 1851 raft_consensus.cc:2229] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:39.626659 1851 raft_consensus.cc:2258] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:39.636878 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 921ac79ba4d94c039b073809e4a17f11 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:39.638299 1851 ts_tablet_manager.cc:1905] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:39.661269 1852 tablet_replica.cc:330] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:39.667241 1849 ts_tablet_manager.cc:1918] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:39.667827 1849 log.cc:1196] T e1fbe5a0e51744568ce535d54093a890 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/e1fbe5a0e51744568ce535d54093a890
I20260806 03:42:39.669867 1849 tablet_replica.cc:330] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:39.669893 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet e1fbe5a0e51744568ce535d54093a890 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:39.670917 1849 raft_consensus.cc:2229] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:39.673593 1849 raft_consensus.cc:2258] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:39.667827 1852 raft_consensus.cc:2229] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:39.676105 1852 raft_consensus.cc:2258] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:39.680298 1852 ts_tablet_manager.cc:1905] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:39.682340 1849 ts_tablet_manager.cc:1905] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:39.689674 1851 ts_tablet_manager.cc:1918] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:39.690192 1851 log.cc:1196] T b19e6696d8d549f58265db62b449e706 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/b19e6696d8d549f58265db62b449e706
I20260806 03:42:39.691898 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet b19e6696d8d549f58265db62b449e706 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:39.700353 1852 ts_tablet_manager.cc:1918] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:39.700995 1852 log.cc:1196] T 921ac79ba4d94c039b073809e4a17f11 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/921ac79ba4d94c039b073809e4a17f11
I20260806 03:42:39.701758 1849 ts_tablet_manager.cc:1918] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:39.702270 1849 log.cc:1196] T 44d06aad0812478bb2270a2c80917306 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/44d06aad0812478bb2270a2c80917306
I20260806 03:42:39.702999 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 921ac79ba4d94c039b073809e4a17f11 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:39.703800 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet 44d06aad0812478bb2270a2c80917306 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
W20260806 03:42:40.173530 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 55976.8us
I20260806 03:42:40.656800 564 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:40.669834 775 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:40.672122 499 consensus_peers.cc:487] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5502493e599c44abbc15f805d5298dbc. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:40.672123 1856 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:40.683331 667 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.686525 1855 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:40.686388 1744 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.694667 775 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.705844 667 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.728084 546 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:40.725546 400 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:40.739063 499 consensus_peers.cc:487] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 760b94c8ff89432c8cbbb7ed61be23b1. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:40.742445 775 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.742609 667 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.744319 1428 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:40.751005 1854 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260806 03:42:40.771428 1744 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.776285 668 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.795456 767 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.796862 775 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:40.816094 546 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.817866 667 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.818609 1857 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:42:40.825014 728 consensus_peers.cc:487] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba8119c361164a62a5ab1232752156bb. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:40.827700 1746 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:40.845635 403 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.858711 1818 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.876791 564 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:40.881505 667 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.893505 400 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:40.904732 546 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:40.977855 667 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:40.977990 499 consensus_peers.cc:487] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b635b355749342d0ad7857884a590b8b. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:40.979393 767 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:40.979872 1428 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:40.981609 1428 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:40.996479 1854 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:41.006024 659 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:41.022751 403 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:41.043253 767 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:41.158212 1887 ts_tablet_manager.cc:927] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:41.163360 1887 tablet_copy_client.cc:316] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:41.165488 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 5502493e599c44abbc15f805d5298dbc from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:41.166127 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 5502493e599c44abbc15f805d5298dbc from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-5502493e599c44abbc15f805d5298dbc
I20260806 03:42:41.181452 575 tablet_copy_source_session.cc:205] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:41.185649 1887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502493e599c44abbc15f805d5298dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:41.203150 1887 tablet_copy_client.cc:791] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:41.203887 1887 tablet_copy_client.cc:655] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:41.210561 1887 tablet_copy_client.cc:531] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:41.222874 1887 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:41.252753 1887 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:41.253790 1887 tablet_bootstrap.cc:492] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:41.254712 1887 ts_tablet_manager.cc:1397] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.013s
I20260806 03:42:41.258044 1887 raft_consensus.cc:348] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.259207 1887 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:41.259882 1887 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.298183 1887 ts_tablet_manager.cc:1428] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.043s user 0.011s sys 0.013s
I20260806 03:42:41.311627 1895 ts_tablet_manager.cc:927] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:41.313706 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-5502493e599c44abbc15f805d5298dbc received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:41.314258 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-5502493e599c44abbc15f805d5298dbc on tablet 5502493e599c44abbc15f805d5298dbc with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:41.317811 1887 ts_tablet_manager.cc:927] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:41.319610 1887 tablet_copy_client.cc:316] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:41.321683 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet ba8119c361164a62a5ab1232752156bb from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:41.322191 1895 tablet_copy_client.cc:316] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:41.322295 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet ba8119c361164a62a5ab1232752156bb from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-ba8119c361164a62a5ab1232752156bb
I20260806 03:42:41.333339 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 760b94c8ff89432c8cbbb7ed61be23b1 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:41.333904 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 760b94c8ff89432c8cbbb7ed61be23b1 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-760b94c8ff89432c8cbbb7ed61be23b1
I20260806 03:42:41.353547 575 tablet_copy_source_session.cc:205] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:41.356012 790 tablet_copy_source_session.cc:205] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:41.358212 1895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 760b94c8ff89432c8cbbb7ed61be23b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:41.359572 1887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8119c361164a62a5ab1232752156bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:41.379272 1887 tablet_copy_client.cc:791] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:41.379944 1887 tablet_copy_client.cc:655] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:41.394292 1895 tablet_copy_client.cc:791] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:41.395012 1895 tablet_copy_client.cc:655] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:41.406535 1894 ts_tablet_manager.cc:927] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:41.409343 1894 tablet_copy_client.cc:316] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:41.410581 1887 tablet_copy_client.cc:531] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:41.414736 1895 tablet_copy_client.cc:531] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:41.416114 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet b635b355749342d0ad7857884a590b8b from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:41.416672 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet b635b355749342d0ad7857884a590b8b from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-b635b355749342d0ad7857884a590b8b
I20260806 03:42:41.427809 575 tablet_copy_source_session.cc:205] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:41.431182 1894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b635b355749342d0ad7857884a590b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:41.436111 1887 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:41.439342 1895 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:41.464308 1894 tablet_copy_client.cc:791] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:41.464951 1894 tablet_copy_client.cc:655] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:41.471041 1895 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:41.471988 1895 tablet_bootstrap.cc:492] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:41.474205 1895 ts_tablet_manager.cc:1397] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.007s
I20260806 03:42:41.476079 1894 tablet_copy_client.cc:531] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:41.487673 1894 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:41.488137 1895 raft_consensus.cc:348] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.489238 1895 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:41.489956 1895 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.493058 1895 ts_tablet_manager.cc:1428] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.018s user 0.000s sys 0.006s
I20260806 03:42:41.494858 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-760b94c8ff89432c8cbbb7ed61be23b1 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:41.495270 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-760b94c8ff89432c8cbbb7ed61be23b1 on tablet 760b94c8ff89432c8cbbb7ed61be23b1 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:41.519654 1887 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:41.520660 1887 tablet_bootstrap.cc:492] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:41.523491 1887 ts_tablet_manager.cc:1397] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.088s user 0.017s sys 0.012s
I20260806 03:42:41.526464 1887 raft_consensus.cc:348] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.527460 1887 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:41.528177 1887 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.534826 1887 ts_tablet_manager.cc:1428] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 03:42:41.536471 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ba8119c361164a62a5ab1232752156bb received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:41.536906 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ba8119c361164a62a5ab1232752156bb on tablet ba8119c361164a62a5ab1232752156bb with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:41.545907 1894 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:41.547050 1894 tablet_bootstrap.cc:492] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:41.551091 1894 ts_tablet_manager.cc:1397] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.064s user 0.019s sys 0.008s
I20260806 03:42:41.554000 1894 raft_consensus.cc:348] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.556216 1894 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:41.556941 1894 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:41.559679 1894 ts_tablet_manager.cc:1428] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:42:41.561363 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-b635b355749342d0ad7857884a590b8b received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:41.561771 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-b635b355749342d0ad7857884a590b8b on tablet b635b355749342d0ad7857884a590b8b with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:41.721817 1671 raft_consensus.cc:1203] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:41.865917 1671 raft_consensus.cc:1203] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:41.888677 1671 raft_consensus.cc:1203] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:41.935519 1671 raft_consensus.cc:1203] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:42.335457 1906 raft_consensus.cc:1050] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:42.338101 1906 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.351089 1671 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.353454 1855 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:42.357007 668 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.359732 775 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.364166 1854 raft_consensus.cc:1050] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:42.366093 1744 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.367736 1854 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.374347 1818 raft_consensus.cc:1050] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:42.374841 1906 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 03:42:42.377252 1818 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.401466 668 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.404546 1668 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.406281 1855 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.409843 1855 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.413044 1668 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.414588 1905 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.451691 1744 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.462270 664 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.466621 546 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.469861 667 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.465862 1818 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.477675 1670 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.485352 1818 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 03:42:42.504144 400 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:42.523859 1906 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.532246 775 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.536978 1668 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.556249 1909 raft_consensus.cc:1050] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:42.561080 1909 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.578675 1855 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:42:42.597108 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 84966.9us
I20260806 03:42:42.613169 1900 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.646721 401 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:42:42.652743 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 77475.5us
W20260806 03:42:42.665077 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 52570.8us
I20260806 03:42:42.670433 667 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.690155 771 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.692399 1668 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.705153 659 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.697661 775 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.743359 767 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.743222 1668 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.751525 1938 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:42.754390 1930 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:42.778685 663 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:42.783741 544 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.800233 401 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.828742 1906 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:42.859293 563 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.861997 546 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.871176 1668 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:42.878232 775 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:42.878480 1906 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:42.885994 1938 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.889098 1671 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.894979 1953 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:42.898669 1854 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.907358 775 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.875054 1670 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:42.874526 667 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:42.920318 1906 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:42.924410 1948 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.943434 771 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:42.942265 767 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.946741 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b635b355749342d0ad7857884a590b8b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:42.958817 400 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.932804 1906 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260806 03:42:42.969766 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5502493e599c44abbc15f805d5298dbc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:42.979738 659 raft_consensus.cc:2932] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.982313 402 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.987253 1671 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:42.989807 1668 raft_consensus.cc:2932] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.000530 1670 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:43.000896 546 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:43.010003 1818 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:43.027007 663 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.030416 1920 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:43.030831 755 tablet_service.cc:1514] Processing DeleteTablet for tablet b635b355749342d0ad7857884a590b8b with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:43.046864 402 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.063551 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502493e599c44abbc15f805d5298dbc with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:43.071030 1818 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.076768 1668 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.107127 1961 tablet_replica.cc:330] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:43.111019 1962 tablet_replica.cc:330] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:43.115883 546 raft_consensus.cc:2932] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.118912 1962 raft_consensus.cc:2229] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:43.117221 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ba8119c361164a62a5ab1232752156bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:43.119566 1962 raft_consensus.cc:2258] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:43.127817 1961 raft_consensus.cc:2229] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:43.128489 1961 raft_consensus.cc:2258] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:43.130015 544 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:43.131531 1962 ts_tablet_manager.cc:1905] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:43.132572 402 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.132158 1961 ts_tablet_manager.cc:1905] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:43.151275 648 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8119c361164a62a5ab1232752156bb with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:43.162257 1668 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:43.169378 1953 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:43.170085 663 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:43.171591 1855 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:43.185107 1938 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.187033 1962 ts_tablet_manager.cc:1918] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:43.187529 1962 log.cc:1196] T 5502493e599c44abbc15f805d5298dbc P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/5502493e599c44abbc15f805d5298dbc
I20260806 03:42:43.189159 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet 5502493e599c44abbc15f805d5298dbc (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:43.191551 1962 tablet_replica.cc:330] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:43.195044 667 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.200508 1668 raft_consensus.cc:2932] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.211369 1962 raft_consensus.cc:2229] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:43.212247 1962 raft_consensus.cc:2258] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:43.212491 1961 ts_tablet_manager.cc:1918] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:43.213063 1961 log.cc:1196] T b635b355749342d0ad7857884a590b8b P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/b635b355749342d0ad7857884a590b8b
I20260806 03:42:43.214942 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet b635b355749342d0ad7857884a590b8b (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:43.216637 1962 ts_tablet_manager.cc:1905] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:43.225055 402 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:43.225986 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 760b94c8ff89432c8cbbb7ed61be23b1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:43.241511 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 760b94c8ff89432c8cbbb7ed61be23b1 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:43.242664 1961 tablet_replica.cc:330] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:43.244010 1961 raft_consensus.cc:2229] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:43.244685 1961 raft_consensus.cc:2258] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:43.248925 1961 ts_tablet_manager.cc:1905] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:43.266809 1961 ts_tablet_manager.cc:1918] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:43.267251 1961 log.cc:1196] T 760b94c8ff89432c8cbbb7ed61be23b1 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/760b94c8ff89432c8cbbb7ed61be23b1
I20260806 03:42:43.269250 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 760b94c8ff89432c8cbbb7ed61be23b1 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:43.269485 1962 ts_tablet_manager.cc:1918] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:43.269920 1962 log.cc:1196] T ba8119c361164a62a5ab1232752156bb P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/ba8119c361164a62a5ab1232752156bb
I20260806 03:42:43.271396 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet ba8119c361164a62a5ab1232752156bb (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:44.311264 659 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.317818 767 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:44.318262 546 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:44.319105 1964 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:44.320169 1830 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 03:42:44.322853 615 consensus_peers.cc:487] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae397e5cb5a14531a20bf02d8c71254d. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:44.328248 1830 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.330014 546 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.336448 767 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.345788 546 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.346820 402 catalog_manager.cc:5495] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.361626 767 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:44.362248 499 consensus_peers.cc:487] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f55827d5ca304ba3a2fb74f4a6e5d1c4. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:44.363425 1854 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:44.364353 659 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:44.365846 1854 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:44.375237 1906 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.384398 767 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.391705 659 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.404210 402 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:44.411170 767 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:44.464764 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 54861.2us
I20260806 03:42:44.472369 659 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:44.473980 546 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:44.477496 1981 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:44.484395 1984 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260806 03:42:44.491161 728 consensus_peers.cc:487] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9369dafb3f342e7a15d44086a377b55. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:44.513967 1982 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.516526 664 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.520455 563 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.533358 664 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.534541 402 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.583657 767 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:44.583731 546 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:44.584417 615 consensus_peers.cc:487] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd5ae69655db414685b926cfecc43d65. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:44.585290 1964 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:44.586189 1954 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:44.597007 1969 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.598424 771 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.607218 546 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.615156 402 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:44.733424 1994 ts_tablet_manager.cc:927] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:44.745208 1994 tablet_copy_client.cc:316] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:42:44.747301 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet ae397e5cb5a14531a20bf02d8c71254d from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:42:44.748150 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet ae397e5cb5a14531a20bf02d8c71254d from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-ae397e5cb5a14531a20bf02d8c71254d
I20260806 03:42:44.765285 691 tablet_copy_source_session.cc:205] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:44.770203 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae397e5cb5a14531a20bf02d8c71254d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:44.784144 1994 tablet_copy_client.cc:791] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:44.784761 1994 tablet_copy_client.cc:655] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:44.788689 1994 tablet_copy_client.cc:531] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:44.797657 1994 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:44.819571 1994 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:44.820541 1994 tablet_bootstrap.cc:492] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:44.821278 1994 ts_tablet_manager.cc:1397] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.013s
I20260806 03:42:44.823956 1994 raft_consensus.cc:348] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.824826 1994 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:44.825454 1994 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.842190 1994 ts_tablet_manager.cc:1428] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
I20260806 03:42:44.845969 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ae397e5cb5a14531a20bf02d8c71254d received from {username='slave'} at 127.0.0.1:46602
I20260806 03:42:44.846344 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ae397e5cb5a14531a20bf02d8c71254d on tablet ae397e5cb5a14531a20bf02d8c71254d with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:44.847856 2000 ts_tablet_manager.cc:927] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:44.856669 2000 tablet_copy_client.cc:316] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:44.858295 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet c9369dafb3f342e7a15d44086a377b55 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:44.858934 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet c9369dafb3f342e7a15d44086a377b55 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-c9369dafb3f342e7a15d44086a377b55
I20260806 03:42:44.865685 790 tablet_copy_source_session.cc:205] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:44.868696 2000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9369dafb3f342e7a15d44086a377b55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:44.883549 2000 tablet_copy_client.cc:791] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:44.884169 2000 tablet_copy_client.cc:655] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:44.888163 2000 tablet_copy_client.cc:531] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:44.897723 2000 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:44.921772 2000 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:44.922662 2000 tablet_bootstrap.cc:492] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:44.923339 2000 ts_tablet_manager.cc:1397] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20260806 03:42:44.926003 2000 raft_consensus.cc:348] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.926900 2000 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:44.927552 2000 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:44.930446 2000 ts_tablet_manager.cc:1428] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:42:44.932778 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-c9369dafb3f342e7a15d44086a377b55 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:44.933205 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-c9369dafb3f342e7a15d44086a377b55 on tablet c9369dafb3f342e7a15d44086a377b55 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:44.945848 2000 ts_tablet_manager.cc:927] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:44.947779 2000 tablet_copy_client.cc:316] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:44.949309 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:44.949808 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-f55827d5ca304ba3a2fb74f4a6e5d1c4
I20260806 03:42:44.957911 575 tablet_copy_source_session.cc:205] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:44.961294 2000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:44.963290 1994 ts_tablet_manager.cc:927] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:44.965036 1994 tablet_copy_client.cc:316] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:42:44.966655 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet dd5ae69655db414685b926cfecc43d65 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:42:44.967124 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet dd5ae69655db414685b926cfecc43d65 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-dd5ae69655db414685b926cfecc43d65
I20260806 03:42:44.975458 691 tablet_copy_source_session.cc:205] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:44.982637 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd5ae69655db414685b926cfecc43d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:44.984583 2000 tablet_copy_client.cc:791] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:44.985285 2000 tablet_copy_client.cc:655] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:45.003585 1994 tablet_copy_client.cc:791] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:45.004194 1994 tablet_copy_client.cc:655] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:45.013108 2000 tablet_copy_client.cc:531] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:45.052116 1994 tablet_copy_client.cc:531] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:45.064564 1994 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:45.074944 2000 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:45.122622 1994 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:45.124482 1994 tablet_bootstrap.cc:492] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:45.125253 1994 ts_tablet_manager.cc:1397] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.061s user 0.022s sys 0.009s
I20260806 03:42:45.131868 2000 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:45.132970 2000 tablet_bootstrap.cc:492] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:45.132424 1994 raft_consensus.cc:348] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:45.133441 1994 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:45.133888 2000 ts_tablet_manager.cc:1397] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.009s
I20260806 03:42:45.135078 1994 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:45.137722 1994 ts_tablet_manager.cc:1428] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20260806 03:42:45.137509 2000 raft_consensus.cc:348] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:45.139631 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-dd5ae69655db414685b926cfecc43d65 received from {username='slave'} at 127.0.0.1:46602
I20260806 03:42:45.140036 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-dd5ae69655db414685b926cfecc43d65 on tablet dd5ae69655db414685b926cfecc43d65 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:45.140650 2000 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:45.141392 2000 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:45.143656 2000 ts_tablet_manager.cc:1428] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:42:45.145329 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-f55827d5ca304ba3a2fb74f4a6e5d1c4 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:45.145691 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-f55827d5ca304ba3a2fb74f4a6e5d1c4 on tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:45.203393 1668 raft_consensus.cc:1203] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260806 03:42:45.255980 1668 raft_consensus.cc:1203] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:45.528659 1668 raft_consensus.cc:1203] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:45.561297 1668 raft_consensus.cc:1203] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:45.633052 1954 raft_consensus.cc:1050] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:45.636859 1954 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.643589 767 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:42:45.643635 1671 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:42:45.644084 546 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:42:45.645009 1830 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:45.646732 1830 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:45.650079 1969 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:42:45.710577 1969 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.715479 767 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.720920 1984 raft_consensus.cc:1050] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:45.724310 1984 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.728971 1671 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.737876 546 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.740329 401 catalog_manager.cc:5495] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.759008 1668 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:45.760679 2014 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:45.767004 663 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:45.767261 544 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:45.769269 1984 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:45.772586 1984 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:45.795356 2014 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.801987 664 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.803905 663 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.803896 544 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.811043 1671 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:42:45.812779 1969 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:45.820475 2017 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.822336 767 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:42:45.824029 1830 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:45.828522 402 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.829738 767 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.828722 1671 raft_consensus.cc:2932] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.840361 1668 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.854941 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ae397e5cb5a14531a20bf02d8c71254d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:45.873307 767 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.887859 563 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:45.891124 1671 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:45.893110 2014 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:45.895519 1984 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:45.884387 403 catalog_manager.cc:5495] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:45.903930 2014 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.908380 1668 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.916652 563 raft_consensus.cc:2932] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.929076 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c9369dafb3f342e7a15d44086a377b55 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:45.936058 536 tablet_service.cc:1514] Processing DeleteTablet for tablet ae397e5cb5a14531a20bf02d8c71254d with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:45.936956 402 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:45.947005 648 tablet_service.cc:1514] Processing DeleteTablet for tablet c9369dafb3f342e7a15d44086a377b55 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:45.959221 2034 tablet_replica.cc:330] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:45.960518 2034 raft_consensus.cc:2229] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:45.951469 1854 raft_consensus.cc:1050] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:45.961352 2034 raft_consensus.cc:2258] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:45.963243 2035 tablet_replica.cc:330] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:45.963474 1969 raft_consensus.cc:1050] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:45.975687 2035 raft_consensus.cc:2229] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:45.976413 2035 raft_consensus.cc:2258] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:45.978063 1969 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.980588 2035 ts_tablet_manager.cc:1905] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:45.965830 1854 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:45.974472 2034 ts_tablet_manager.cc:1905] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:45.995803 1668 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:45.997846 1830 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:46.003031 767 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:46.004333 1969 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:46.004829 546 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:46.007198 1830 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:46.032226 1830 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.032809 1671 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:46.036670 2035 ts_tablet_manager.cc:1918] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:46.037362 767 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:46.040148 2034 ts_tablet_manager.cc:1918] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260806 03:42:46.040234 2024 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:46.040632 2034 log.cc:1196] T ae397e5cb5a14531a20bf02d8c71254d P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/ae397e5cb5a14531a20bf02d8c71254d
I20260806 03:42:46.041138 2035 log.cc:1196] T c9369dafb3f342e7a15d44086a377b55 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/c9369dafb3f342e7a15d44086a377b55
I20260806 03:42:46.041345 1668 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.044252 2024 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:42:46.046253 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet ae397e5cb5a14531a20bf02d8c71254d (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:46.048084 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet c9369dafb3f342e7a15d44086a377b55 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:46.059221 667 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:46.058687 771 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.065600 1854 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:46.076123 400 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:46.077354 544 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.100382 1854 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.107769 667 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:46.114230 1670 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.124317 771 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.139832 1668 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:46.141201 1830 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:46.136587 402 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:46.147879 1969 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.155550 767 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:46.155300 1670 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.159235 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dd5ae69655db414685b926cfecc43d65 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:46.160271 1830 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:46.156214 664 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.164369 767 raft_consensus.cc:2932] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.171830 400 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:46.188558 563 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:46.198813 1670 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:46.199457 536 tablet_service.cc:1514] Processing DeleteTablet for tablet dd5ae69655db414685b926cfecc43d65 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:46.200518 2034 tablet_replica.cc:330] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:46.200313 2024 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:46.201491 2034 raft_consensus.cc:2229] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:46.203240 2034 raft_consensus.cc:2258] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:46.204566 663 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:46.206859 2034 ts_tablet_manager.cc:1905] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:46.209641 2012 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:46.226049 1854 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.228209 2034 ts_tablet_manager.cc:1918] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:46.228679 2034 log.cc:1196] T dd5ae69655db414685b926cfecc43d65 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/dd5ae69655db414685b926cfecc43d65
I20260806 03:42:46.230710 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet dd5ae69655db414685b926cfecc43d65 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:46.232936 663 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.235658 1670 raft_consensus.cc:2932] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:46.239537 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:46.244132 400 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:46.258939 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:46.277858 2049 tablet_replica.cc:330] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:46.278985 2049 raft_consensus.cc:2229] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:46.279693 2049 raft_consensus.cc:2258] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:46.284607 2049 ts_tablet_manager.cc:1905] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:46.304317 2049 ts_tablet_manager.cc:1918] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:46.304767 2049 log.cc:1196] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/f55827d5ca304ba3a2fb74f4a6e5d1c4
I20260806 03:42:46.306177 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet f55827d5ca304ba3a2fb74f4a6e5d1c4 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:47.314291 767 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.322733 663 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.325165 2053 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:47.325435 544 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.327126 2050 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 03:42:47.327785 728 consensus_peers.cc:487] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2a414fbc8cc4236a2f89a5bec69c6c2. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:47.340307 2050 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.342175 663 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.355103 544 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.366070 563 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.380049 400 catalog_manager.cc:5495] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:47.399557 767 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.400748 663 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:47.403908 499 consensus_peers.cc:487] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bea9147cd344c6d8e771c5d9968f2e7. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:47.404222 2012 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:47.405954 2056 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:47.419979 2007 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.421624 663 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.439067 563 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.443133 775 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
W20260806 03:42:47.486395 499 consensus_peers.cc:487] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25056084174744caba1a79c3eb002598. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:47.487553 663 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.487970 775 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.489428 2012 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:47.481081 402 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.498135 2058 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:47.523044 2056 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.525449 663 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.525502 775 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.542110 402 catalog_manager.cc:5495] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:47.544541 771 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:47.564061 728 consensus_peers.cc:487] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d2a8e791d4943d9a81d8d8c75d09783. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:47.567003 663 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.567577 544 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:47.568331 1986 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:47.569221 2050 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:47.577731 2053 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.582983 544 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.584439 663 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.601504 403 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:47.863600 2076 ts_tablet_manager.cc:927] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:47.865727 2076 tablet_copy_client.cc:316] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:47.867307 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:47.867856 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet c2a414fbc8cc4236a2f89a5bec69c6c2 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-c2a414fbc8cc4236a2f89a5bec69c6c2
I20260806 03:42:47.875655 790 tablet_copy_source_session.cc:205] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:47.878938 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a414fbc8cc4236a2f89a5bec69c6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:47.902760 2076 tablet_copy_client.cc:791] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:47.903488 2076 tablet_copy_client.cc:655] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:47.907267 2076 tablet_copy_client.cc:531] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:47.914891 2082 ts_tablet_manager.cc:927] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:47.916980 2082 tablet_copy_client.cc:316] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:47.918679 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 2bea9147cd344c6d8e771c5d9968f2e7 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:47.919169 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 2bea9147cd344c6d8e771c5d9968f2e7 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-2bea9147cd344c6d8e771c5d9968f2e7
I20260806 03:42:47.927934 575 tablet_copy_source_session.cc:205] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:47.929673 2076 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:47.931612 2082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bea9147cd344c6d8e771c5d9968f2e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:47.948330 2082 tablet_copy_client.cc:791] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:47.949051 2082 tablet_copy_client.cc:655] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:47.953012 2076 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:47.953970 2076 tablet_bootstrap.cc:492] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:47.954700 2076 ts_tablet_manager.cc:1397] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.011s
I20260806 03:42:47.954903 2082 tablet_copy_client.cc:531] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:47.957634 2076 raft_consensus.cc:348] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.958571 2076 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:47.959182 2076 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.965122 2082 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:47.976347 2076 ts_tablet_manager.cc:1428] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.021s user 0.008s sys 0.012s
I20260806 03:42:47.978188 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-c2a414fbc8cc4236a2f89a5bec69c6c2 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:47.978636 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-c2a414fbc8cc4236a2f89a5bec69c6c2 on tablet c2a414fbc8cc4236a2f89a5bec69c6c2 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:47.990224 2082 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:47.991333 2082 tablet_bootstrap.cc:492] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:47.991983 2082 ts_tablet_manager.cc:1397] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.027s user 0.021s sys 0.004s
I20260806 03:42:47.994547 2082 raft_consensus.cc:348] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.995450 2082 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:47.996109 2082 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:47.998626 2082 ts_tablet_manager.cc:1428] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 03:42:48.002981 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-2bea9147cd344c6d8e771c5d9968f2e7 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:48.003393 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-2bea9147cd344c6d8e771c5d9968f2e7 on tablet 2bea9147cd344c6d8e771c5d9968f2e7 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:48.009346 2082 ts_tablet_manager.cc:927] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:48.025933 2082 tablet_copy_client.cc:316] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:48.027377 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 25056084174744caba1a79c3eb002598 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:48.027956 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 25056084174744caba1a79c3eb002598 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-25056084174744caba1a79c3eb002598
I20260806 03:42:48.036252 575 tablet_copy_source_session.cc:205] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:48.041074 2082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25056084174744caba1a79c3eb002598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:48.058629 2082 tablet_copy_client.cc:791] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:48.059258 2082 tablet_copy_client.cc:655] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:48.065709 2082 tablet_copy_client.cc:531] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:48.082487 2082 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:48.102442 2082 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:48.103360 2082 tablet_bootstrap.cc:492] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:48.103955 2082 ts_tablet_manager.cc:1397] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20260806 03:42:48.106621 2082 raft_consensus.cc:348] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:48.107439 2082 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:48.108265 2082 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:48.113406 2082 ts_tablet_manager.cc:1428] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 03:42:48.115432 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-25056084174744caba1a79c3eb002598 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:48.115815 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-25056084174744caba1a79c3eb002598 on tablet 25056084174744caba1a79c3eb002598 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:48.152736 2082 ts_tablet_manager.cc:927] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:48.154515 2082 tablet_copy_client.cc:316] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:48.156091 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet 9d2a8e791d4943d9a81d8d8c75d09783 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:48.156563 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet 9d2a8e791d4943d9a81d8d8c75d09783 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-9d2a8e791d4943d9a81d8d8c75d09783
I20260806 03:42:48.164000 790 tablet_copy_source_session.cc:205] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:48.166944 2082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d2a8e791d4943d9a81d8d8c75d09783. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:48.184249 2082 tablet_copy_client.cc:791] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:48.184815 2082 tablet_copy_client.cc:655] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:48.189091 2082 tablet_copy_client.cc:531] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:48.198124 2082 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:48.217837 2082 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:48.218722 2082 tablet_bootstrap.cc:492] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:48.219343 2082 ts_tablet_manager.cc:1397] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.004s
I20260806 03:42:48.223055 2082 raft_consensus.cc:348] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:48.223869 2082 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:48.224730 2082 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:48.230674 2082 ts_tablet_manager.cc:1428] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260806 03:42:48.233551 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-9d2a8e791d4943d9a81d8d8c75d09783 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:48.234210 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-9d2a8e791d4943d9a81d8d8c75d09783 on tablet 9d2a8e791d4943d9a81d8d8c75d09783 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:48.276937 1670 raft_consensus.cc:1203] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:48.418963 1670 raft_consensus.cc:1203] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:48.435286 1670 raft_consensus.cc:1203] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:48.686509 2053 raft_consensus.cc:1050] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:48.689029 2053 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:48.707306 1670 raft_consensus.cc:1203] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:48.721179 1671 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.722913 544 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.723265 664 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.723786 2053 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:48.727123 2050 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:48.728163 2098 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:48.752882 2050 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.754099 1670 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.758556 667 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.769692 563 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.774586 403 catalog_manager.cc:5495] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:48.811767 2012 raft_consensus.cc:1050] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:48.811489 767 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:48.819062 667 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:48.820132 1671 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:48.821561 2053 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:48.828258 2053 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:48.832553 2056 raft_consensus.cc:1050] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:48.833527 2053 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.835024 2056 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:48.814167 2012 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:48.836232 667 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.846027 1670 raft_consensus.cc:2932] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.859750 1671 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.860335 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:48.859762 767 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.861567 667 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.861254 2095 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:48.862924 2097 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 03:42:48.866894 402 catalog_manager.cc:5495] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.892489 667 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.896109 1671 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.880927 2058 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260806 03:42:48.901003 771 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:48.904321 2012 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:48.918936 2095 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260806 03:42:48.939924 536 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a414fbc8cc4236a2f89a5bec69c6c2 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:48.938560 2058 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260806 03:42:48.979597 2012 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.982918 2118 tablet_replica.cc:330] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:48.983817 664 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.989421 2112 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.993824 1670 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:48.998637 2118 raft_consensus.cc:2229] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:48.999522 2118 raft_consensus.cc:2258] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:49.004032 2118 ts_tablet_manager.cc:1905] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:49.012264 664 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.009894 1671 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.044898 400 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.056108 767 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.059041 771 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.073828 2118 ts_tablet_manager.cc:1918] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:49.083840 401 catalog_manager.cc:5495] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.096166 2118 log.cc:1196] T c2a414fbc8cc4236a2f89a5bec69c6c2 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/c2a414fbc8cc4236a2f89a5bec69c6c2
I20260806 03:42:49.098392 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet c2a414fbc8cc4236a2f89a5bec69c6c2 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:49.110627 563 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:49.110772 544 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:49.120496 1670 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.121656 1671 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.122053 664 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.123090 2111 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:49.123461 2012 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:49.126226 771 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.125876 2111 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:49.134863 2095 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:49.133620 2056 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.141757 664 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.144598 1670 raft_consensus.cc:2932] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.151676 2095 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.153594 1671 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.161418 771 raft_consensus.cc:2932] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.168537 401 catalog_manager.cc:5495] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:49.172092 403 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.174135 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 25056084174744caba1a79c3eb002598 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:49.197252 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2bea9147cd344c6d8e771c5d9968f2e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:49.210752 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bea9147cd344c6d8e771c5d9968f2e7 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:49.223304 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 25056084174744caba1a79c3eb002598 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:49.223387 2122 tablet_replica.cc:330] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:49.237730 2122 raft_consensus.cc:2229] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:49.238744 2122 raft_consensus.cc:2258] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:49.242383 2123 tablet_replica.cc:330] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:49.244026 2123 raft_consensus.cc:2229] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:49.248553 2123 raft_consensus.cc:2258] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:49.247695 2122 ts_tablet_manager.cc:1905] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:49.263545 2123 ts_tablet_manager.cc:1905] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:49.290755 2122 ts_tablet_manager.cc:1918] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:49.293776 2122 log.cc:1196] T 2bea9147cd344c6d8e771c5d9968f2e7 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/2bea9147cd344c6d8e771c5d9968f2e7
I20260806 03:42:49.296453 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet 2bea9147cd344c6d8e771c5d9968f2e7 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:49.297746 2123 ts_tablet_manager.cc:1918] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:49.298504 2123 log.cc:1196] T 25056084174744caba1a79c3eb002598 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/25056084174744caba1a79c3eb002598
I20260806 03:42:49.301260 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 25056084174744caba1a79c3eb002598 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:49.320348 2098 raft_consensus.cc:1050] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:49.323956 2098 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:49.333472 664 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:49.333807 544 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:49.334170 1671 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:49.334800 2098 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:49.336714 2053 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:49.338768 2050 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:49.377933 2050 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.382243 544 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.383720 1671 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.400014 402 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.410445 664 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.430440 771 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:49.436579 664 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.438525 1671 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:49.438345 2050 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:49.442868 2050 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:49.450266 2098 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.452293 667 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.468331 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9d2a8e791d4943d9a81d8d8c75d09783 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:49.478871 400 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:49.489470 1668 raft_consensus.cc:2932] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:49.503373 536 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d2a8e791d4943d9a81d8d8c75d09783 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:49.504678 2118 tablet_replica.cc:330] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:49.505779 2118 raft_consensus.cc:2229] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:49.506804 2118 raft_consensus.cc:2258] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:49.511549 2118 ts_tablet_manager.cc:1905] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:49.531258 2118 ts_tablet_manager.cc:1918] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:49.531909 2118 log.cc:1196] T 9d2a8e791d4943d9a81d8d8c75d09783 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/9d2a8e791d4943d9a81d8d8c75d09783
I20260806 03:42:49.533588 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet 9d2a8e791d4943d9a81d8d8c75d09783 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:50.571179 564 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:50.579186 771 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.579185 667 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:50.581146 499 consensus_peers.cc:487] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fdb062ea0544f0ab7202b067aaa662e. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:50.582636 2130 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.583787 2138 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.594689 2130 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.596632 771 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.596655 667 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.617604 400 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.627990 546 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:50.638051 499 consensus_peers.cc:487] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8a4c59f28ea45f6998c3f7f0b7dda53. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:50.642973 664 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.644685 2111 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.656457 771 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.663923 2056 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.687994 2130 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.703537 767 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.710273 667 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.735913 767 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:50.741276 402 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:50.751557 659 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:50.753258 728 consensus_peers.cc:487] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a9afce5efa4491797ebed2aab67532f. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:50.754762 2098 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.757058 546 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.758482 2140 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:50.775274 2133 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.780396 546 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.782444 667 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.793831 771 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:50.797871 402 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:50.805886 728 consensus_peers.cc:487] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68a9e5190380411684128482e3a899b8. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:50.806939 667 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.808847 546 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:50.812091 2098 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:50.813755 2140 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:50.823160 2133 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.824967 564 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.834571 667 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:50.845835 402 catalog_manager.cc:5495] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:51.009557 2163 ts_tablet_manager.cc:927] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:51.015208 2163 tablet_copy_client.cc:316] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:51.017143 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 1fdb062ea0544f0ab7202b067aaa662e from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:51.017702 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 1fdb062ea0544f0ab7202b067aaa662e from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-1fdb062ea0544f0ab7202b067aaa662e
I20260806 03:42:51.025583 575 tablet_copy_source_session.cc:205] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:51.035550 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fdb062ea0544f0ab7202b067aaa662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:51.053015 2163 tablet_copy_client.cc:791] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:51.053829 2163 tablet_copy_client.cc:655] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:51.059258 2163 tablet_copy_client.cc:531] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:51.068344 2163 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:51.075572 2165 ts_tablet_manager.cc:927] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:51.078068 2165 tablet_copy_client.cc:316] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:51.082297 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:51.083164 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet e8a4c59f28ea45f6998c3f7f0b7dda53 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-e8a4c59f28ea45f6998c3f7f0b7dda53
I20260806 03:42:51.097450 2163 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:51.098532 2163 tablet_bootstrap.cc:492] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:51.099323 2163 ts_tablet_manager.cc:1397] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.011s
I20260806 03:42:51.102710 2163 raft_consensus.cc:348] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.103817 2163 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:51.104190 575 tablet_copy_source_session.cc:205] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:51.104554 2163 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.107641 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a4c59f28ea45f6998c3f7f0b7dda53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:51.109130 2163 ts_tablet_manager.cc:1428] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260806 03:42:51.111150 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-1fdb062ea0544f0ab7202b067aaa662e received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:51.111629 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-1fdb062ea0544f0ab7202b067aaa662e on tablet 1fdb062ea0544f0ab7202b067aaa662e with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:51.130191 2165 tablet_copy_client.cc:791] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:51.130880 2165 tablet_copy_client.cc:655] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:51.135103 2165 tablet_copy_client.cc:531] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:51.144411 2165 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:51.171190 2165 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:51.172180 2165 tablet_bootstrap.cc:492] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:51.172950 2165 ts_tablet_manager.cc:1397] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.011s
I20260806 03:42:51.175729 2165 raft_consensus.cc:348] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.176599 2165 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:51.177263 2165 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.179589 2165 ts_tablet_manager.cc:1428] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 03:42:51.181469 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-e8a4c59f28ea45f6998c3f7f0b7dda53 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:51.181945 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-e8a4c59f28ea45f6998c3f7f0b7dda53 on tablet e8a4c59f28ea45f6998c3f7f0b7dda53 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:51.202425 2165 ts_tablet_manager.cc:927] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:51.204695 2165 tablet_copy_client.cc:316] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:51.206684 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet 7a9afce5efa4491797ebed2aab67532f from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:51.207227 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet 7a9afce5efa4491797ebed2aab67532f from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-7a9afce5efa4491797ebed2aab67532f
I20260806 03:42:51.225230 790 tablet_copy_source_session.cc:205] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:51.228889 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9afce5efa4491797ebed2aab67532f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:51.245047 2165 tablet_copy_client.cc:791] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:51.245730 2165 tablet_copy_client.cc:655] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:51.250916 2165 tablet_copy_client.cc:531] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:51.259558 2163 ts_tablet_manager.cc:927] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:51.259647 2165 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:51.262110 2163 tablet_copy_client.cc:316] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:51.263768 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet 68a9e5190380411684128482e3a899b8 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:51.264335 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet 68a9e5190380411684128482e3a899b8 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-68a9e5190380411684128482e3a899b8
I20260806 03:42:51.281858 790 tablet_copy_source_session.cc:205] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:51.282553 2165 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:51.283572 2165 tablet_bootstrap.cc:492] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:51.284289 2165 ts_tablet_manager.cc:1397] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.004s
I20260806 03:42:51.285161 2163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68a9e5190380411684128482e3a899b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:51.287335 2165 raft_consensus.cc:348] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.288357 2165 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:51.289057 2165 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.291865 2165 ts_tablet_manager.cc:1428] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:42:51.298481 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-7a9afce5efa4491797ebed2aab67532f received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:51.300163 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-7a9afce5efa4491797ebed2aab67532f on tablet 7a9afce5efa4491797ebed2aab67532f with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:51.321619 2163 tablet_copy_client.cc:791] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:51.322270 2163 tablet_copy_client.cc:655] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:51.325848 2163 tablet_copy_client.cc:531] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:51.334486 2163 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:51.356746 2163 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:51.357977 2163 tablet_bootstrap.cc:492] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:51.358992 2163 ts_tablet_manager.cc:1397] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.008s
I20260806 03:42:51.363826 2163 raft_consensus.cc:348] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.365170 2163 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:51.366051 2163 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:51.370877 2163 ts_tablet_manager.cc:1428] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260806 03:42:51.374892 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-68a9e5190380411684128482e3a899b8 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:51.375439 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-68a9e5190380411684128482e3a899b8 on tablet 68a9e5190380411684128482e3a899b8 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:51.463099 1671 raft_consensus.cc:1203] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:51.589798 1671 raft_consensus.cc:1203] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:51.669690 1671 raft_consensus.cc:1203] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:51.694262 1671 raft_consensus.cc:1203] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:51.867515 2179 raft_consensus.cc:1050] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:51.870350 2179 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:51.888123 1671 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:51.888206 659 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:51.889518 2130 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:51.892059 775 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:51.892871 2179 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:51.894860 2130 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:51.904832 2056 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.906512 768 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.911907 667 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.924773 401 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:51.927481 1671 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.962903 564 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:51.970611 1668 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:51.971524 775 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:51.972283 2130 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:51.973264 2056 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:51.979992 2136 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.981352 1668 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.984663 768 raft_consensus.cc:2932] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:51.999498 400 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.001312 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:52.034732 648 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a4c59f28ea45f6998c3f7f0b7dda53 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:52.037585 2196 tablet_replica.cc:330] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:52.053198 2196 raft_consensus.cc:2229] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:52.054039 2196 raft_consensus.cc:2258] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:52.057626 2196 ts_tablet_manager.cc:1905] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:52.073732 2196 ts_tablet_manager.cc:1918] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:52.074177 2196 log.cc:1196] T e8a4c59f28ea45f6998c3f7f0b7dda53 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/e8a4c59f28ea45f6998c3f7f0b7dda53
I20260806 03:42:52.076164 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet e8a4c59f28ea45f6998c3f7f0b7dda53 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:52.155849 2180 raft_consensus.cc:1050] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:52.158203 2180 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.174775 1671 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.174762 667 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.175808 563 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.176656 2133 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.177718 2098 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.181260 2180 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:52.194208 2133 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.194995 563 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.203529 2056 raft_consensus.cc:1050] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:52.202817 667 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.207796 1671 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.214154 400 catalog_manager.cc:5495] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.205894 2056 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.229607 1671 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.230073 659 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.232182 768 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.232039 2130 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.234747 2130 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.237855 2136 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.280177 768 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.286732 1671 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.294926 663 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.319811 2053 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:52.321981 2053 raft_consensus.cc:1050] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:52.320917 2136 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.323428 663 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.328526 2053 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.333040 2180 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.338186 1668 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.377679 1668 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.393551 403 catalog_manager.cc:5495] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.397022 402 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:42:52.419800 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 64509.3us
I20260806 03:42:52.424805 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 68a9e5190380411684128482e3a899b8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:52.417209 775 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:42:52.431586 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 66528us
I20260806 03:42:52.439043 547 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.441079 667 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.442399 2213 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:52.446053 1671 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:52.454478 2212 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 03:42:52.457377 2098 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.473919 536 tablet_service.cc:1514] Processing DeleteTablet for tablet 68a9e5190380411684128482e3a899b8 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:52.480758 2098 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:52.487849 564 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.494817 663 raft_consensus.cc:2932] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.501151 659 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.504890 1671 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.513018 2179 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:52.514395 2056 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:52.524531 2209 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.529853 1670 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.538347 659 raft_consensus.cc:2932] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.557938 2222 tablet_replica.cc:330] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:52.571352 2222 raft_consensus.cc:2229] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:52.572222 2222 raft_consensus.cc:2258] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:52.574146 401 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.606685 2222 ts_tablet_manager.cc:1905] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:52.609330 2133 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.615087 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb062ea0544f0ab7202b067aaa662e with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:52.637456 1670 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.639644 664 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.643538 2222 ts_tablet_manager.cc:1918] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:52.644234 2222 log.cc:1196] T 68a9e5190380411684128482e3a899b8 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/68a9e5190380411684128482e3a899b8
I20260806 03:42:52.645363 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1fdb062ea0544f0ab7202b067aaa662e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:52.649928 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet 68a9e5190380411684128482e3a899b8 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:52.661870 403 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:52.692927 771 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:52.695510 563 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.737717 2234 tablet_replica.cc:330] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:52.745626 1670 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.748917 547 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:52.756318 2212 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:52.756983 2234 raft_consensus.cc:2229] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:52.757848 2234 raft_consensus.cc:2258] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:52.758949 2212 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:52.764928 2213 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.766558 547 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.775111 1670 raft_consensus.cc:2932] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.778858 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7a9afce5efa4491797ebed2aab67532f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:52.784704 400 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:52.791944 2234 ts_tablet_manager.cc:1905] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:52.804615 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9afce5efa4491797ebed2aab67532f with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:52.806704 2236 tablet_replica.cc:330] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:52.807907 2236 raft_consensus.cc:2229] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:52.808933 2236 raft_consensus.cc:2258] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:52.813012 2236 ts_tablet_manager.cc:1905] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:52.823462 2234 ts_tablet_manager.cc:1918] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:52.824018 2234 log.cc:1196] T 1fdb062ea0544f0ab7202b067aaa662e P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/1fdb062ea0544f0ab7202b067aaa662e
I20260806 03:42:52.826416 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 1fdb062ea0544f0ab7202b067aaa662e (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:52.830547 2236 ts_tablet_manager.cc:1918] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:52.830964 2236 log.cc:1196] T 7a9afce5efa4491797ebed2aab67532f P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/7a9afce5efa4491797ebed2aab67532f
I20260806 03:42:52.832329 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet 7a9afce5efa4491797ebed2aab67532f (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:53.863994 563 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:53.870169 499 consensus_peers.cc:487] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 604c561db3f3435da75d2536381f6b96. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:53.870332 771 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.870591 667 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.871676 2237 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:53.872630 2130 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:53.888513 2237 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.889889 667 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.890985 768 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.907500 400 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.915671 563 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:53.927829 499 consensus_peers.cc:487] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce3b590a95fa411a91b9a3818bf7d29f. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:53.930864 768 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.931789 667 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.932801 2241 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:53.934233 2136 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:53.959233 2240 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.962330 664 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.969684 768 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.981443 775 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:53.982517 401 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:53.990933 664 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.992158 563 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:53.992764 2133 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:53.998081 2212 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 03:42:54.006750 728 consensus_peers.cc:487] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fdbf67cdfa6f4f2785238e161fd2c4c8. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:54.011745 2264 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.012820 664 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.020406 547 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.027521 768 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.030805 403 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.041563 547 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:54.042924 2212 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 03:42:54.044343 728 consensus_peers.cc:487] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d8ac3ebccff4054953475f839218e45. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:54.044773 664 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:54.046937 2212 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:54.056061 2264 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.058225 547 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.058955 667 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.078110 401 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:54.262102 2268 ts_tablet_manager.cc:927] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:54.264308 2268 tablet_copy_client.cc:316] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:54.266026 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 604c561db3f3435da75d2536381f6b96 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:54.266588 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 604c561db3f3435da75d2536381f6b96 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-604c561db3f3435da75d2536381f6b96
I20260806 03:42:54.275789 575 tablet_copy_source_session.cc:205] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:54.278609 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 604c561db3f3435da75d2536381f6b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:54.296541 2268 tablet_copy_client.cc:791] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:54.297223 2268 tablet_copy_client.cc:655] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:54.301779 2268 tablet_copy_client.cc:531] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:54.311740 2268 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:54.339807 2271 ts_tablet_manager.cc:927] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:54.339807 2268 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:54.341084 2268 tablet_bootstrap.cc:492] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:54.342022 2268 ts_tablet_manager.cc:1397] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.030s user 0.017s sys 0.013s
I20260806 03:42:54.345953 2268 raft_consensus.cc:348] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.347414 2268 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:54.348727 2271 tablet_copy_client.cc:316] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:54.348279 2268 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.351491 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet ce3b590a95fa411a91b9a3818bf7d29f from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:54.351648 2268 ts_tablet_manager.cc:1428] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20260806 03:42:54.352077 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet ce3b590a95fa411a91b9a3818bf7d29f from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-ce3b590a95fa411a91b9a3818bf7d29f
I20260806 03:42:54.358899 573 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-604c561db3f3435da75d2536381f6b96 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:54.359387 573 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-604c561db3f3435da75d2536381f6b96 on tablet 604c561db3f3435da75d2536381f6b96 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:54.361122 575 tablet_copy_source_session.cc:205] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:54.364619 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3b590a95fa411a91b9a3818bf7d29f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:54.380332 2271 tablet_copy_client.cc:791] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:54.380980 2271 tablet_copy_client.cc:655] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:54.386198 2271 tablet_copy_client.cc:531] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:54.398485 2271 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:54.420233 2271 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:54.421329 2271 tablet_bootstrap.cc:492] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:54.422040 2271 ts_tablet_manager.cc:1397] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.012s
I20260806 03:42:54.424935 2271 raft_consensus.cc:348] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.425789 2271 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:54.426560 2271 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.429069 2271 ts_tablet_manager.cc:1428] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260806 03:42:54.431209 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ce3b590a95fa411a91b9a3818bf7d29f received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:54.431622 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ce3b590a95fa411a91b9a3818bf7d29f on tablet ce3b590a95fa411a91b9a3818bf7d29f with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:54.565049 2271 ts_tablet_manager.cc:927] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:54.566861 2271 tablet_copy_client.cc:316] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:54.569362 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:54.569911 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet fdbf67cdfa6f4f2785238e161fd2c4c8 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-fdbf67cdfa6f4f2785238e161fd2c4c8
I20260806 03:42:54.577728 790 tablet_copy_source_session.cc:205] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:54.581619 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdbf67cdfa6f4f2785238e161fd2c4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:54.597646 2271 tablet_copy_client.cc:791] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:54.598253 2271 tablet_copy_client.cc:655] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:54.602509 2271 tablet_copy_client.cc:531] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:54.611228 2271 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:54.647806 2271 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:54.648823 2271 tablet_bootstrap.cc:492] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:54.649674 2271 ts_tablet_manager.cc:1397] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.024s
I20260806 03:42:54.651291 2268 ts_tablet_manager.cc:927] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:54.652757 2271 raft_consensus.cc:348] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.654088 2271 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:54.655095 2271 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.658245 2271 ts_tablet_manager.cc:1428] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260806 03:42:54.662034 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-fdbf67cdfa6f4f2785238e161fd2c4c8 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:54.662505 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-fdbf67cdfa6f4f2785238e161fd2c4c8 on tablet fdbf67cdfa6f4f2785238e161fd2c4c8 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:54.666443 2268 tablet_copy_client.cc:316] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:54.669124 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet 3d8ac3ebccff4054953475f839218e45 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:54.669724 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet 3d8ac3ebccff4054953475f839218e45 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-3d8ac3ebccff4054953475f839218e45
I20260806 03:42:54.678006 790 tablet_copy_source_session.cc:205] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:54.683945 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8ac3ebccff4054953475f839218e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:54.700529 2268 tablet_copy_client.cc:791] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:54.701200 2268 tablet_copy_client.cc:655] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:54.722492 2268 tablet_copy_client.cc:531] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:54.736160 2268 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:54.753870 2268 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:54.754803 2268 tablet_bootstrap.cc:492] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:54.755506 2268 ts_tablet_manager.cc:1397] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.011s
I20260806 03:42:54.758315 2268 raft_consensus.cc:348] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.760113 2268 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:54.760741 2268 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:54.763161 2268 ts_tablet_manager.cc:1428] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260806 03:42:54.765755 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-3d8ac3ebccff4054953475f839218e45 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:54.766192 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-3d8ac3ebccff4054953475f839218e45 on tablet 3d8ac3ebccff4054953475f839218e45 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:54.825402 1670 raft_consensus.cc:1203] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:54.887264 1670 raft_consensus.cc:1203] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:55.096267 1670 raft_consensus.cc:1203] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:55.188868 1670 raft_consensus.cc:1203] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:55.261961 2241 raft_consensus.cc:1050] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:55.264410 2241 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.271587 1668 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.272439 768 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.273319 2241 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.274704 667 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.275888 2284 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.276810 2240 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.297227 2240 raft_consensus.cc:1050] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:55.300377 2240 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.306516 2284 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.308892 771 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.316007 1668 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.330130 667 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.334796 775 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.334354 400 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.337925 2241 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.339521 664 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.341006 2136 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.351729 1670 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.353152 2241 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.367902 2292 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.375227 1668 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.375113 667 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.380156 775 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.411904 564 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.419055 667 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.417105 403 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.421094 2136 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:55.432549 2241 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.435097 663 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.439028 1668 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.440384 2292 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:55.445369 1670 raft_consensus.cc:2932] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.465329 402 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.473922 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ce3b590a95fa411a91b9a3818bf7d29f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:55.477777 563 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.483304 1670 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.485435 2284 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:55.487015 663 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.488408 2303 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:55.492702 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3b590a95fa411a91b9a3818bf7d29f with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:55.493052 2284 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.498423 667 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.498317 1670 raft_consensus.cc:2932] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.506423 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 604c561db3f3435da75d2536381f6b96 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:55.510788 2304 tablet_replica.cc:330] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:55.512120 2304 raft_consensus.cc:2229] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:55.512844 2304 raft_consensus.cc:2258] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:55.518568 2304 ts_tablet_manager.cc:1905] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:55.524281 400 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:55.543865 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 604c561db3f3435da75d2536381f6b96 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:55.547091 2304 ts_tablet_manager.cc:1918] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:55.547691 2304 log.cc:1196] T ce3b590a95fa411a91b9a3818bf7d29f P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/ce3b590a95fa411a91b9a3818bf7d29f
I20260806 03:42:55.549654 2304 tablet_replica.cc:330] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:55.550186 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet ce3b590a95fa411a91b9a3818bf7d29f (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:55.550866 2304 raft_consensus.cc:2229] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:55.551693 2304 raft_consensus.cc:2258] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:55.555241 2304 ts_tablet_manager.cc:1905] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:55.571334 2304 ts_tablet_manager.cc:1918] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:55.571762 2304 log.cc:1196] T 604c561db3f3435da75d2536381f6b96 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/604c561db3f3435da75d2536381f6b96
I20260806 03:42:55.573436 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 604c561db3f3435da75d2536381f6b96 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:55.617434 2262 raft_consensus.cc:1050] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:55.621614 2262 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.628381 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:55.633708 1670 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.636353 667 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.636447 2212 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.638228 2212 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:55.638926 563 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.640677 2286 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:55.664525 2212 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.672050 564 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.679106 667 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.698455 401 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.671057 1670 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.718954 775 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.730690 667 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.732039 2212 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:55.738397 1668 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:55.745714 2318 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.745819 2262 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:55.753069 775 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:55.755738 1668 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.771639 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:55.773836 775 raft_consensus.cc:595] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:55.757000 2212 raft_consensus.cc:1050] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:55.777395 2212 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:55.778537 771 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:55.779400 771 raft_consensus.cc:595] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:55.792024 663 raft_consensus.cc:2932] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:55.836982 401 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:55.839339 767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:55.848820 767 raft_consensus.cc:595] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:55.853308 767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:55.854305 767 raft_consensus.cc:595] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:55.869055 564 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.869860 664 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.872378 1668 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:55.872376 2262 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:42:55.876612 767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:55.877218 767 raft_consensus.cc:595] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:55.879009 536 tablet_service.cc:1514] Processing DeleteTablet for tablet fdbf67cdfa6f4f2785238e161fd2c4c8 with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:42:55.886806 2318 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:55.912277 2325 tablet_replica.cc:330] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:42:55.916666 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:55.917284 564 raft_consensus.cc:595] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:55.919154 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:42:55.920853 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:55.921536 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:55.913578 2325 raft_consensus.cc:2229] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:55.932551 2325 raft_consensus.cc:2258] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 03:42:55.933734 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 63277.6us
I20260806 03:42:55.964632 2325 ts_tablet_manager.cc:1905] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:55.971606 2324 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:55.976420 2262 raft_consensus.cc:979] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:55.977005 2262 raft_consensus.cc:1067] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:55.991930 667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:42:55.992545 667 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:55.992906 667 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.000715 667 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.003700 2262 raft_consensus.cc:979] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:56.004529 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.005270 563 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.008358 2262 raft_consensus.cc:1067] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:56.010944 1670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60246
I20260806 03:42:56.011462 1670 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:56.011775 1670 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.012601 2325 ts_tablet_manager.cc:1918] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:56.013007 2325 log.cc:1196] T fdbf67cdfa6f4f2785238e161fd2c4c8 P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/fdbf67cdfa6f4f2785238e161fd2c4c8
I20260806 03:42:56.016095 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet fdbf67cdfa6f4f2785238e161fd2c4c8 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:56.017818 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.018553 775 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.018575 563 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.018982 775 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:56.019896 775 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.021296 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:56.022104 2320 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.022594 2320 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:56.023432 2320 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.023619 1670 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.014564 667 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.029897 1670 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.032050 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.032732 767 raft_consensus.cc:3032] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.033030 767 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:56.033753 767 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.036041 767 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.039628 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.040577 563 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.047276 2262 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.048810 775 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.056139 767 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:56.057528 1624 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:42:56.058640 2274 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.061728 564 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.071969 563 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:56.074174 2274 raft_consensus.cc:686] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:56.075336 2274 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.076402 775 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.072724 401 catalog_manager.cc:5495] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:56.082329 2286 raft_consensus.cc:979] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.082810 2318 raft_consensus.cc:1067] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.087069 663 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.089579 2317 raft_consensus.cc:979] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.090071 2317 raft_consensus.cc:1067] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.088287 667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:42:56.091636 667 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:56.092090 667 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.094717 664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:42:56.095324 664 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:56.095639 664 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.098680 402 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.103389 1670 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.118973 767 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.122836 664 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.126514 1668 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:56.129803 2318 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:56.132838 664 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.134052 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.134703 767 raft_consensus.cc:3032] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.134991 767 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:56.135811 767 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.129040 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.136896 767 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.137144 563 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.142673 667 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.144943 400 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:56.147158 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.147773 775 raft_consensus.cc:3032] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.148072 775 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:56.148592 2286 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.150110 667 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.149807 775 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.151017 775 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.151937 563 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.153622 614 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:56.154656 2320 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.155369 2320 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:56.157644 775 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.157452 2320 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.155773 1668 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.163947 767 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.176273 563 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:42:56.177542 616 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:56.178176 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3d8ac3ebccff4054953475f839218e45 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:56.178484 2221 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.178995 2221 raft_consensus.cc:686] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:56.179356 2324 raft_consensus.cc:979] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.179829 2324 raft_consensus.cc:1067] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:56.179860 2221 consensus_queue.cc:237] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.183933 663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:42:56.184489 663 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:56.184832 663 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.196795 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.197492 564 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.195655 663 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.204025 401 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:56.206089 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.207152 563 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.213001 2286 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:56.229662 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8ac3ebccff4054953475f839218e45 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:42:56.230656 563 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.221030 547 raft_consensus.cc:2932] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:56.231874 614 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:56.228332 564 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.233208 2320 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.233896 2320 raft_consensus.cc:686] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:56.234771 2320 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.238912 2338 tablet_replica.cc:330] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:42:56.241667 2338 raft_consensus.cc:2229] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:56.242784 2338 raft_consensus.cc:2258] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:56.246703 2338 ts_tablet_manager.cc:1905] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:56.247699 663 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.247920 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.248510 767 raft_consensus.cc:3032] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.248811 767 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:56.249595 767 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:56.251027 767 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.257722 767 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:56.263482 401 catalog_manager.cc:5495] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:56.265170 401 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:56.266520 2338 ts_tablet_manager.cc:1918] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:42:56.267036 2338 log.cc:1196] T 3d8ac3ebccff4054953475f839218e45 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/3d8ac3ebccff4054953475f839218e45
I20260806 03:42:56.268751 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet 3d8ac3ebccff4054953475f839218e45 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:56.273823 2303 raft_consensus.cc:979] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:56.274478 2303 raft_consensus.cc:1067] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:56.276067 1668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60258
I20260806 03:42:56.276585 1668 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:56.276924 1668 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.282613 1668 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.282644 401 catalog_manager.cc:5495] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:56.285346 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:56.285745 1668 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:56.285866 564 raft_consensus.cc:3032] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:56.286345 564 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:56.287091 564 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.288278 564 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.294934 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:56.295670 767 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:56.301950 564 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:56.303721 1622 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:56.304638 2274 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:56.305124 2274 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:56.306432 2274 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:56.311713 767 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:56.323014 401 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:56.460246 564 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.461752 2320 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:56.477077 767 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.479229 2323 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:56.565245 564 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.566756 2323 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:56.576822 767 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.578217 2131 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:56.654130 564 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:56.655457 2274 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:56.662935 768 raft_consensus.cc:1261] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.664535 2131 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:56.664819 768 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:56.666602 2274 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:56.695663 547 raft_consensus.cc:1261] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.697443 2323 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:56.704175 564 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.705844 2323 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:56.723587 768 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:56.725138 2131 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:56.793186 547 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:56.794706 2274 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:56.806532 768 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:56.808023 2364 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:57.005784 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:57.044382 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:57.045065 564 raft_consensus.cc:595] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:57.048324 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:57.048900 564 raft_consensus.cc:595] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:57.051155 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:57.051546 564 raft_consensus.cc:595] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:57.053526 768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:57.053905 768 raft_consensus.cc:595] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:42:57.055882 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:57.056272 564 raft_consensus.cc:595] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:57.057107 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:42:57.058347 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:57.058944 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:57.086184 2366 raft_consensus.cc:979] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:57.086787 2241 raft_consensus.cc:1067] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:57.088183 768 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:42:57.088976 768 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:57.089408 768 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.097546 2303 raft_consensus.cc:979] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:57.097184 768 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.098084 2366 raft_consensus.cc:1067] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:57.099556 1670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60258
I20260806 03:42:57.100103 1670 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:57.100430 1670 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.100759 768 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:57.101650 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:57.101650 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:57.102262 667 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.102397 547 raft_consensus.cc:3032] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:57.102818 547 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:57.103500 547 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.104508 547 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.109844 1670 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:57.110491 667 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:57.111979 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:57.112820 2324 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:57.113464 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:57.114023 775 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.115373 547 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:57.115818 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:57.116353 564 raft_consensus.cc:3032] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:57.116624 564 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:57.117331 564 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:57.118083 564 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.120528 775 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:57.121759 1624 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:42:57.122654 2364 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:57.114145 2324 raft_consensus.cc:686] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:57.123117 2364 raft_consensus.cc:686] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:57.123899 2324 consensus_queue.cc:237] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.124029 2364 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:57.130497 1670 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:57.133939 564 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:57.143872 401 catalog_manager.cc:5495] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:57.147814 402 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:57.164975 2262 raft_consensus.cc:979] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:57.165622 2324 raft_consensus.cc:1067] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:57.167402 667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:42:57.168115 667 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:57.168696 667 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.186803 667 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.189029 667 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:57.189289 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:57.189894 547 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.190486 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:57.191018 775 raft_consensus.cc:3032] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:57.191309 775 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:57.192031 775 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.193209 775 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.197575 547 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:57.199055 614 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:57.199844 2131 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:57.200254 2131 raft_consensus.cc:686] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:57.200971 2131 consensus_queue.cc:237] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.201514 775 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:57.218528 401 catalog_manager.cc:5495] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:57.240860 2303 raft_consensus.cc:979] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:57.241358 2241 raft_consensus.cc:1067] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:42:57.242655 667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:48530
I20260806 03:42:57.243101 667 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:57.243361 667 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.248968 667 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.251407 667 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:57.251391 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:57.252257 547 raft_consensus.cc:3032] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:57.252547 547 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:57.253096 547 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.254218 547 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.257396 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:57.257969 775 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.259830 547 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:57.260769 614 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:57.262149 2131 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:57.262743 2131 raft_consensus.cc:686] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:57.264243 775 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:42:57.267577 2131 consensus_queue.cc:237] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.275447 775 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:57.283362 663 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:57.279600 401 catalog_manager.cc:5495] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:42:57.284576 728 consensus_peers.cc:487] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edcb8ef1a79d4601b243ec9ce6e6acb7. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:57.285707 547 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:57.287860 2324 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.289808 2324 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:57.307156 2262 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.309998 547 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.311919 663 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.333387 401 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.335556 667 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:42:57.343065 615 consensus_peers.cc:487] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 329fbbb627464f478b8d2fd7a28851cc. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:57.344751 775 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:42:57.346230 547 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:42:57.346204 2320 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:57.352166 2221 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:57.353976 2221 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.355428 775 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.359952 564 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.376169 667 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:57.379024 400 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:57.384351 564 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:42:57.384477 615 consensus_peers.cc:487] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e65c896c1a47c4a0a7eb7b1036bbba. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:57.384397 775 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:57.386708 2221 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.387638 2131 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.399078 2320 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.418090 775 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.417341 564 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.437028 547 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:57.438324 401 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:42:57.455909 667 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:57.457122 768 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:42:57.457685 2391 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260806 03:42:57.458810 499 consensus_peers.cc:487] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 736f0260dd3a4e77a6115e55fa0c4269. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:42:57.458942 2241 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.468412 2366 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.469995 667 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.470593 768 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.487637 400 catalog_manager.cc:5495] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:42:57.573320 2241 raft_consensus.cc:979] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:57.573772 2241 raft_consensus.cc:1067] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:57.575354 768 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:42:57.577399 768 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:57.578526 768 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.582535 547 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.583755 2320 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.585625 768 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.590031 768 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:57.591573 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:57.591601 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:57.592252 564 raft_consensus.cc:3032] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:57.592309 663 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.592629 564 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:57.593341 564 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.594338 564 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:57.600162 564 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:57.602205 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:42:57.603122 2262 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:57.603673 2262 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:57.604835 2262 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:57.601871 663 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:57.640908 401 catalog_manager.cc:5495] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:57.644707 768 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.645421 775 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:57.648106 2334 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:57.646602 2221 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260806 03:42:57.683985 564 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.685664 2131 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.692176 659 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.698850 2262 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:57.707298 547 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:57.710135 2334 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:57.721694 564 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.737869 2381 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260806 03:42:57.784811 2418 ts_tablet_manager.cc:927] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:57.786834 767 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:57.792256 2221 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:57.796913 2418 tablet_copy_client.cc:316] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:42:57.799224 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet d1e65c896c1a47c4a0a7eb7b1036bbba from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:42:57.799986 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet d1e65c896c1a47c4a0a7eb7b1036bbba from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-d1e65c896c1a47c4a0a7eb7b1036bbba
W20260806 03:42:57.839811 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 48930.5us
I20260806 03:42:57.872924 691 tablet_copy_source_session.cc:205] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:57.923700 2424 ts_tablet_manager.cc:927] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:57.925199 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e65c896c1a47c4a0a7eb7b1036bbba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:57.928495 2431 ts_tablet_manager.cc:927] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:57.933288 2431 tablet_copy_client.cc:316] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:42:57.935272 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet 329fbbb627464f478b8d2fd7a28851cc from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:42:57.936010 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet 329fbbb627464f478b8d2fd7a28851cc from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-329fbbb627464f478b8d2fd7a28851cc
W20260806 03:42:57.939244 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 100619us
I20260806 03:42:57.943095 2424 tablet_copy_client.cc:316] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.67:43937
I20260806 03:42:57.944713 790 tablet_copy_service.cc:140] P 0374a84d5e9f4e3c8d8287e2382379c8: Received BeginTabletCopySession request for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46314)
I20260806 03:42:57.945212 790 tablet_copy_service.cc:161] P 0374a84d5e9f4e3c8d8287e2382379c8: Beginning new tablet copy session on tablet edcb8ef1a79d4601b243ec9ce6e6acb7 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46314: session id = 1914b652c9fb49b694d71a5f5f5752ee-edcb8ef1a79d4601b243ec9ce6e6acb7
W20260806 03:42:57.955503 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 92677.7us
I20260806 03:42:57.956143 790 tablet_copy_source_session.cc:205] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:57.961222 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edcb8ef1a79d4601b243ec9ce6e6acb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:57.987980 2418 tablet_copy_client.cc:791] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:57.988698 2418 tablet_copy_client.cc:655] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:57.994571 2418 tablet_copy_client.cc:531] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:57.994789 691 tablet_copy_source_session.cc:205] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:58.012570 2424 tablet_copy_client.cc:791] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:58.021839 2424 tablet_copy_client.cc:655] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:58.018975 2436 ts_tablet_manager.cc:927] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:42:58.020705 2431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 329fbbb627464f478b8d2fd7a28851cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:58.038605 2424 tablet_copy_client.cc:531] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:58.058354 2431 tablet_copy_client.cc:791] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:58.059134 2436 tablet_copy_client.cc:316] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:42:58.059497 2431 tablet_copy_client.cc:655] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:58.061952 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 736f0260dd3a4e77a6115e55fa0c4269 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:42:58.062422 2424 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:58.062764 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 736f0260dd3a4e77a6115e55fa0c4269 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-736f0260dd3a4e77a6115e55fa0c4269
I20260806 03:42:58.071938 2431 tablet_copy_client.cc:531] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:58.087750 2418 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:58.092106 575 tablet_copy_source_session.cc:205] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:42:58.097666 2431 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:58.098845 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:58.099936 2436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 736f0260dd3a4e77a6115e55fa0c4269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:42:58.137710 563 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:58.140590 2381 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:58.147609 667 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:58.149338 2262 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:42:58.320591 2436 tablet_copy_client.cc:791] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:42:58.327759 2436 tablet_copy_client.cc:655] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:42:58.343719 2436 tablet_copy_client.cc:531] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:42:58.351214 2424 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:58.352897 659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d06aad0812478bb2270a2c80917306"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:42:58.353603 659 raft_consensus.cc:595] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.353929 2424 tablet_bootstrap.cc:492] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:58.354836 2424 ts_tablet_manager.cc:1397] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.293s user 0.020s sys 0.033s
I20260806 03:42:58.358914 2424 raft_consensus.cc:348] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.360019 2424 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:58.360430 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "497d006930504e8184f1e5e512ade026"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:58.361411 544 raft_consensus.cc:595] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:58.361009 2424 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.364998 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "604c561db3f3435da75d2536381f6b96"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:58.365614 544 raft_consensus.cc:595] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.369508 2424 ts_tablet_manager.cc:1428] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20260806 03:42:58.371526 790 tablet_copy_service.cc:342] P 0374a84d5e9f4e3c8d8287e2382379c8: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-edcb8ef1a79d4601b243ec9ce6e6acb7 received from {username='slave'} at 127.0.0.1:46314
I20260806 03:42:58.372185 790 tablet_copy_service.cc:433] P 0374a84d5e9f4e3c8d8287e2382379c8: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-edcb8ef1a79d4601b243ec9ce6e6acb7 on tablet edcb8ef1a79d4601b243ec9ce6e6acb7 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.378975 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:58.379927 544 raft_consensus.cc:595] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.383181 768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:58.383940 768 raft_consensus.cc:595] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.384578 2436 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:42:58.387224 768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:42:58.387918 768 raft_consensus.cc:595] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.389140 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:42:58.394945 2431 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:58.398690 2431 tablet_bootstrap.cc:492] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:58.399873 2431 ts_tablet_manager.cc:1397] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.303s user 0.038s sys 0.016s
I20260806 03:42:58.403086 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:42:58.405175 2418 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:58.410868 2431 raft_consensus.cc:348] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.412448 2431 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:58.413343 2431 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.417285 2431 ts_tablet_manager.cc:1428] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.017s user 0.007s sys 0.007s
I20260806 03:42:58.418599 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:58.419605 2418 tablet_bootstrap.cc:492] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:58.420442 2418 ts_tablet_manager.cc:1397] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.333s user 0.023s sys 0.021s
I20260806 03:42:58.420581 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-329fbbb627464f478b8d2fd7a28851cc received from {username='slave'} at 127.0.0.1:46602
I20260806 03:42:58.421000 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-329fbbb627464f478b8d2fd7a28851cc on tablet 329fbbb627464f478b8d2fd7a28851cc with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.425689 2418 raft_consensus.cc:348] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.427142 2418 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:58.436592 2418 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.445878 2418 ts_tablet_manager.cc:1428] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.025s user 0.005s sys 0.005s
I20260806 03:42:58.448099 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-d1e65c896c1a47c4a0a7eb7b1036bbba received from {username='slave'} at 127.0.0.1:46602
I20260806 03:42:58.448670 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-d1e65c896c1a47c4a0a7eb7b1036bbba on tablet d1e65c896c1a47c4a0a7eb7b1036bbba with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.472985 2436 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:42:58.473989 2436 tablet_bootstrap.cc:492] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:42:58.474912 2436 ts_tablet_manager.cc:1397] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.095s user 0.033s sys 0.008s
I20260806 03:42:58.478185 2436 raft_consensus.cc:348] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.479279 2436 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:42:58.480340 2436 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:42:58.484298 2436 ts_tablet_manager.cc:1428] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260806 03:42:58.488909 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-736f0260dd3a4e77a6115e55fa0c4269 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:42:58.489478 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-736f0260dd3a4e77a6115e55fa0c4269 on tablet 736f0260dd3a4e77a6115e55fa0c4269 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:58.503719 1670 raft_consensus.cc:1203] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:42:58.567318 2320 raft_consensus.cc:979] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.568032 2320 raft_consensus.cc:1067] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.569772 1670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44d06aad0812478bb2270a2c80917306"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60272
I20260806 03:42:58.570446 1670 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.570760 1670 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.581161 1670 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.584359 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.584970 659 raft_consensus.cc:3032] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.585238 659 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:42:58.586004 1670 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:58.585996 659 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.587685 659 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.587658 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:58.588460 775 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.594686 659 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.595687 775 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.595973 1622 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:58.597738 2334 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.598208 2334 raft_consensus.cc:686] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:58.599152 2334 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.609669 402 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.694953 2430 raft_consensus.cc:979] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.695462 2432 raft_consensus.cc:1067] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.697618 1671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "604c561db3f3435da75d2536381f6b96"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60258
I20260806 03:42:58.698154 1671 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.698510 1671 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.704422 1671 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.705886 1671 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:58.706843 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.706831 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:58.707623 667 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.707793 547 raft_consensus.cc:3032] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.708133 547 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:58.708937 547 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.710005 547 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.714170 667 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.715399 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:58.716295 2334 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.716809 2334 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:58.717587 547 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.717675 2334 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.737922 2381 raft_consensus.cc:979] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.738476 2324 raft_consensus.cc:1067] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.740658 1671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60246
I20260806 03:42:58.741303 1671 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.741376 402 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.742249 1671 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.744336 2380 raft_consensus.cc:979] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.745121 2423 raft_consensus.cc:1067] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.747673 1670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60246
I20260806 03:42:58.748277 1670 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.748716 1670 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.756377 1671 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.758987 1671 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:58.759833 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.759949 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:58.760478 667 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.760572 768 raft_consensus.cc:3032] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.760934 768 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:58.761509 768 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.762609 768 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.762969 1670 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.765439 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.765970 659 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.767094 768 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.767144 667 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.768296 1624 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:42:58.769269 2364 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.769761 2364 raft_consensus.cc:686] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:58.770810 2364 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.780503 659 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.781749 1622 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:58.782954 2460 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.783653 2460 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:58.784489 2460 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.786239 1670 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:42:58.786298 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:42:58.787019 768 raft_consensus.cc:3032] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.787318 768 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:58.788085 768 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.789295 768 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.796303 768 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.802968 1670 raft_consensus.cc:1203] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:58.818518 2430 raft_consensus.cc:979] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.820978 2430 raft_consensus.cc:1067] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:42:58.822816 1671 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60258
I20260806 03:42:58.823575 1671 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.823997 1671 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.830876 402 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.842494 1670 raft_consensus.cc:1203] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:58.846079 1671 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.855480 1671 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:58.856555 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:58.856961 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.857326 547 raft_consensus.cc:3032] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.857617 659 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.857664 547 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:58.858592 547 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.861420 547 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.866165 659 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.873152 2241 raft_consensus.cc:979] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:58.873684 2241 raft_consensus.cc:1067] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:42:58.873896 1622 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:58.875588 775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "497d006930504e8184f1e5e512ade026"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:42:58.876181 775 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:42:58.876579 775 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.886183 2364 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.887601 2364 raft_consensus.cc:686] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:42:58.888681 2364 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.901624 547 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:42:58.903051 775 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:58.906297 775 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:42:58.912036 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:42:58.913975 664 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.913331 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:42:58.918866 547 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:42:58.919358 547 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:42:58.921437 664 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:58.922675 727 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:42:58.922899 547 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:58.924067 2380 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:42:58.924232 547 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:42:58.924788 2380 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:42:58.925712 2380 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:42:58.932986 547 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:42:58.933446 1670 raft_consensus.cc:1203] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:42:58.941515 2401 raft_consensus.cc:1050] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:58.944236 2401 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:58.945957 400 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.947875 400 catalog_manager.cc:5495] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.957863 402 catalog_manager.cc:5495] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:42:58.964527 1670 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:42:58.966234 2221 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:58.972184 547 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:42:58.973062 775 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:42:58.973992 2401 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:42:58.976475 2221 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:42:58.992597 2401 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:58.993870 767 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.012979 1670 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.016883 400 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.017628 547 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.059087 664 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:59.059489 667 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.060808 2460 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:59.064780 1670 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.065976 547 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.067822 2320 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:59.068816 767 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.070693 2401 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:59.070099 2460 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.080215 2131 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.089797 544 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.089069 1670 raft_consensus.cc:2932] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.114114 402 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.126098 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 329fbbb627464f478b8d2fd7a28851cc with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:42:59.129408 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 329fbbb627464f478b8d2fd7a28851cc with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47796
I20260806 03:42:59.154493 2481 tablet_replica.cc:330] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:42:59.174603 2481 raft_consensus.cc:2229] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:42:59.175555 2481 raft_consensus.cc:2258] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:42:59.178807 2481 ts_tablet_manager.cc:1905] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:42:59.183928 775 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.186513 2364 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.198344 2481 ts_tablet_manager.cc:1918] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 03:42:59.199015 2481 log.cc:1196] T 329fbbb627464f478b8d2fd7a28851cc P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/329fbbb627464f478b8d2fd7a28851cc
I20260806 03:42:59.203248 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 329fbbb627464f478b8d2fd7a28851cc (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:42:59.209348 664 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.211297 2334 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.228922 663 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.230604 2334 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.234797 667 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.236295 2334 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.250110 768 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.251834 2460 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.271152 547 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.280645 2334 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:59.315693 667 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.323457 2334 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:42:59.347494 544 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:42:59.350438 2334 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:42:59.371577 2131 raft_consensus.cc:1050] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:59.373893 2131 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:59.391018 775 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:59.392422 1671 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:59.392817 2320 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:59.394201 2221 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:42:59.404018 2366 raft_consensus.cc:1050] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:59.417160 2366 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:59.434237 544 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:59.452143 2320 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.470712 564 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:59.471383 663 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:42:59.477785 2324 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:59.463001 1671 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.465930 2380 raft_consensus.cc:1050] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:42:59.479627 2493 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:42:59.483201 2380 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:42:59.484694 2131 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:42:59.503144 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:42:59.523716 767 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.533495 1671 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:42:59.522784 564 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.539860 401 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:42:59.556675 2432 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
W20260806 03:42:59.710292 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 174555us
W20260806 03:42:59.652287 2501 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 03:42:59.730995 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:59.731604 544 raft_consensus.cc:595] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:42:59.735863 1668 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 03:42:59.671381 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56197.5us
W20260806 03:42:59.737268 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 128076us
I20260806 03:42:59.740975 664 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:42:59.780504 2504 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260806 03:42:59.786149 1670 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260806 03:42:59.811743 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 80930.5us
I20260806 03:42:59.868466 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:59.869807 564 raft_consensus.cc:595] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:42:59.842834 2513 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260806 03:42:59.914355 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:42:59.915278 564 raft_consensus.cc:595] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
W20260806 03:42:59.950974 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 180450us
W20260806 03:43:00.032658 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 261787us
I20260806 03:43:00.070668 767 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:00.072748 768 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:00.071599 767 raft_consensus.cc:595] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:00.124315 663 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:00.125134 663 raft_consensus.cc:595] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:00.125546 663 raft_consensus.cc:619] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Rejecting request to transfer leadership because tablet server ada5ef6656cb4420baf77a4b4342909e is not a voter in the active config
I20260806 03:43:00.121440 2380 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
W20260806 03:43:00.105831 2500 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 03:43:00.145629 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:43:00.147912 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:43:00.104568 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 252652us
I20260806 03:43:00.148751 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:43:00.185055 2366 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.008s sys 0.017s
W20260806 03:43:00.185518 2366 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.670s user 0.009s sys 0.017s
I20260806 03:43:00.193490 663 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 03:43:00.289899 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 249342us
W20260806 03:43:00.282444 2498 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
W20260806 03:43:00.352943 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 156080us
I20260806 03:43:00.360682 2541 raft_consensus.cc:979] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:43:00.366317 2324 raft_consensus.cc:979] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:00.371824 2463 raft_consensus.cc:1067] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:43:00.497954 2508 raft_consensus.cc:1067] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
W20260806 03:43:00.508652 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 262707us
W20260806 03:43:00.516441 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 73225.9us
I20260806 03:43:00.527228 2562 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20260806 03:43:00.518764 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 54551.9us
I20260806 03:43:00.560846 769 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 03:43:00.566406 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 139386us
I20260806 03:43:00.568797 2533 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:00.567480 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 242744us
I20260806 03:43:00.578765 1664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60258
I20260806 03:43:00.579771 1664 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:00.580291 1664 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.587342 1670 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:00.665541 2432 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:00.663169 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 59718.9us
I20260806 03:43:00.666332 2432 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:00.673169 2432 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:00.677460 1664 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:00.692097 2366 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:00.703379 667 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:00.707649 400 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New cstate: current_term: 1 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:00.707832 2366 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:00.710865 544 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:00.711122 663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:43:00.711920 663 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:00.712394 663 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.715760 2366 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:00.720202 2366 raft_consensus.cc:979] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:00.720868 2366 raft_consensus.cc:1067] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:00.722330 2221 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:00.723070 2221 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:00.725124 2221 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:00.726770 1670 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:00.729686 663 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:00.732309 2503 raft_consensus.cc:979] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:00.740710 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:00.741641 671 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.750469 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d1e65c896c1a47c4a0a7eb7b1036bbba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:00.729012 1664 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:00.757603 2503 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:00.758878 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:00.760861 2503 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:00.762858 2538 raft_consensus.cc:1067] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:00.772982 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:00.774166 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:00.775247 769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:43:00.775980 769 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:00.776423 769 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.782532 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:00.783320 663 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:00.785989 1670 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.794029 775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:43:00.794803 775 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:00.795197 775 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.798172 2517 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260806 03:43:00.813283 664 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:00.822564 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:00.820402 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:00.826350 2558 consensus_queue.cc:1046] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260806 03:43:00.827296 2508 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:00.832257 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:00.843428 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:00.844476 771 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:00.843529 2558 raft_consensus.cc:3032] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:00.845072 771 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:00.856591 498 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:00.859439 1670 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:00.850400 2558 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:00.860292 1671 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:00.864019 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:00.866187 2221 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:00.864341 2558 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:00.872335 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:00.846534 771 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:00.875774 2517 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:00.874477 2558 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.849988 775 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:00.879496 2506 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:00.888302 770 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:00.890149 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:00.891873 775 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:00.900748 671 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:00.902546 2515 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 03:43:00.903784 2538 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:00.924255 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:00.943360 674 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:00.948047 771 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.955307 2432 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:00.956061 2432 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:00.957480 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:00.959146 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:00.961577 665 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:00.964967 2221 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:00.965615 2221 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:00.985347 2221 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:01.005921 2366 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:00.990541 2432 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:01.017972 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:01.031158 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 93052us
I20260806 03:43:01.031402 2535 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:01.032032 2535 raft_consensus.cc:686] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:01.006268 769 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:01.029389 2366 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:01.066563 776 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.069147 2432 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:01.083622 2558 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.085956 2366 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:01.033252 2535 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:01.106611 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:01.111788 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:01.117357 771 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:01.119994 564 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
W20260806 03:43:01.121954 2570 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.072s user 0.001s sys 0.001s
I20260806 03:43:01.129295 772 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:01.131880 769 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:01.132790 498 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:01.133663 2515 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:01.143679 547 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:01.149361 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:01.157814 665 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:01.149437 2515 consensus_queue.cc:1046] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260806 03:43:01.167943 2515 raft_consensus.cc:3032] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:01.168503 2515 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:01.149664 772 raft_consensus.cc:2932] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.157073 674 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.169935 2515 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:01.180328 2515 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:01.191382 2574 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
W20260806 03:43:01.205202 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 114162us
I20260806 03:43:01.206636 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:01.208350 2508 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:01.208880 2508 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:01.211242 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:01.211709 2516 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.215821 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:01.216562 664 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:01.209796 2508 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:01.220628 1671 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.228217 564 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:01.226223 558 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.234014 664 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:01.249022 727 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:01.236649 665 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.250423 2576 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:01.251164 2576 raft_consensus.cc:686] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:01.252074 2576 consensus_queue.cc:237] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:01.329523 401 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.337515 1671 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.343477 536 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e65c896c1a47c4a0a7eb7b1036bbba with delete_type TABLET_DATA_TOMBSTONED (TS ada5ef6656cb4420baf77a4b4342909e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35200
I20260806 03:43:01.369781 402 catalog_manager.cc:5495] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.377609 402 catalog_manager.cc:5495] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:01.380585 402 catalog_manager.cc:5495] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:01.411623 400 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:01.448441 1671 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:01.452215 2562 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260806 03:43:01.511536 769 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:01.526139 2595 tablet_replica.cc:330] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: stopping tablet replica
I20260806 03:43:01.529812 1671 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:01.530723 562 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:01.533362 2519 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:01.535151 2519 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:01.544003 2423 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.545104 772 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:01.546444 1664 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.552065 558 raft_consensus.cc:2932] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.560387 378 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:01.572649 402 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:01.618148 648 tablet_service.cc:1514] Processing DeleteTablet for tablet edcb8ef1a79d4601b243ec9ce6e6acb7 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:43:01.620522 2562 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:01.622622 400 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:01.628764 2595 raft_consensus.cc:2229] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:01.632521 2595 raft_consensus.cc:2258] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:01.637923 2595 ts_tablet_manager.cc:1905] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:43:01.639815 2602 tablet_replica.cc:330] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:43:01.641206 2602 raft_consensus.cc:2229] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:01.642138 2602 raft_consensus.cc:2258] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:01.647743 2602 ts_tablet_manager.cc:1905] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:43:01.683182 664 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:01.684651 2535 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:01.695055 558 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:01.697328 2535 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:01.727451 562 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:01.736045 1664 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:01.738188 2554 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:01.749087 2602 ts_tablet_manager.cc:1918] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:43:01.752935 2602 log.cc:1196] T edcb8ef1a79d4601b243ec9ce6e6acb7 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/edcb8ef1a79d4601b243ec9ce6e6acb7
I20260806 03:43:01.752595 772 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:01.752557 2558 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.757699 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet edcb8ef1a79d4601b243ec9ce6e6acb7 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:43:01.760605 2554 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:01.766089 769 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.770082 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 736f0260dd3a4e77a6115e55fa0c4269 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:01.751526 2595 ts_tablet_manager.cc:1918] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:43:01.771193 2595 log.cc:1196] T d1e65c896c1a47c4a0a7eb7b1036bbba P ada5ef6656cb4420baf77a4b4342909e: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-0-root/wals/d1e65c896c1a47c4a0a7eb7b1036bbba
I20260806 03:43:01.773396 1664 raft_consensus.cc:2932] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.791888 664 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:01.796175 376 catalog_manager.cc:4841] TS ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): tablet d1e65c896c1a47c4a0a7eb7b1036bbba (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:43:01.798614 2517 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:01.822633 400 catalog_manager.cc:5495] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:01.827706 558 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:01.831683 2516 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:01.844770 648 tablet_service.cc:1514] Processing DeleteTablet for tablet 736f0260dd3a4e77a6115e55fa0c4269 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:43:01.845881 2602 tablet_replica.cc:330] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:43:01.847088 2602 raft_consensus.cc:2229] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:01.847971 2602 raft_consensus.cc:2258] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:01.852507 2602 ts_tablet_manager.cc:1905] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:43:01.859797 664 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:01.861702 2517 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:01.876447 2602 ts_tablet_manager.cc:1918] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:43:01.877064 2602 log.cc:1196] T 736f0260dd3a4e77a6115e55fa0c4269 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/736f0260dd3a4e77a6115e55fa0c4269
I20260806 03:43:01.882807 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:01.891753 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet 736f0260dd3a4e77a6115e55fa0c4269 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:43:01.893381 544 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:01.895035 2517 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:01.963774 664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:01.964550 664 raft_consensus.cc:595] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:01.968098 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:01.968693 544 raft_consensus.cc:595] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:01.971735 664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:01.972287 664 raft_consensus.cc:595] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Received request to transfer leadership to TS 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:01.975425 769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:01.976037 769 raft_consensus.cc:595] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:01.977375 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:43:01.978899 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:01.979586 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:02.059396 2558 raft_consensus.cc:979] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:02.060021 2554 raft_consensus.cc:1067] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:02.061970 769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:43:02.062572 769 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:02.062839 769 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.064000 2517 raft_consensus.cc:979] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:02.064522 2576 raft_consensus.cc:1067] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:02.065987 544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:35214
I20260806 03:43:02.066512 544 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:02.066890 544 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.072188 769 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:02.074175 544 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.075572 769 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:02.076681 544 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:02.076902 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:02.077487 1670 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.077982 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:02.078594 544 raft_consensus.cc:3032] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:02.078683 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:02.078918 544 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:02.079319 664 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.079792 544 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:02.081177 544 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.085136 1670 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:02.087393 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:02.088276 2554 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:02.088776 2554 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:02.089295 544 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:02.089933 2554 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.094554 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:02.095216 769 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:02.095494 769 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:02.095937 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:02.096263 769 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.096802 2576 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:02.097555 769 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.098469 2576 raft_consensus.cc:686] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:02.100031 2562 raft_consensus.cc:979] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:43:02.101310 2562 raft_consensus.cc:1067] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Signalling peer 1914b652c9fb49b694d71a5f5f5752ee to start an election
I20260806 03:43:02.103181 1664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
from {username='slave'} at 127.0.0.1:60272
I20260806 03:43:02.103866 1664 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:02.104251 1664 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:02.108001 769 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:02.101732 401 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:02.111557 1664 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.114428 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:02.115072 665 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:02.115396 665 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:02.116217 665 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.117200 665 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:02.117794 664 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:02.118989 1664 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:02.120080 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:02.120736 772 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:02.114459 2576 consensus_queue.cc:237] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:02.127841 772 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:02.130424 665 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:02.131609 1622 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:02.132437 2561 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:02.132949 2561 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:02.133831 2561 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:02.136518 401 catalog_manager.cc:5495] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:02.141649 400 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 3, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 3 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:02.263993 2221 raft_consensus.cc:979] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:02.264510 2575 raft_consensus.cc:1067] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:02.266597 558 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:37194
I20260806 03:43:02.267338 558 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:02.267813 558 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.275843 558 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:02.278656 558 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:02.279161 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:02.279889 665 raft_consensus.cc:3032] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:02.280200 665 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:02.281258 665 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:02.282562 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:02.283733 772 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.284085 665 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:02.291366 772 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:02.292764 501 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:02.293929 2432 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:02.294453 2432 raft_consensus.cc:686] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:02.295348 2432 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:02.305305 665 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:02.309077 400 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 1 to 2, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:02.568511 776 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:02.570299 2554 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:02.583838 1664 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:02.585229 2554 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:02.595654 665 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:02.597518 2561 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:02.609961 776 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:02.611833 558 raft_consensus.cc:1261] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:02.613327 2561 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:02.613803 2517 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:02.648274 665 raft_consensus.cc:1261] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:02.649735 2517 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:02.824259 665 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:02.825675 2432 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:02.842135 776 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:02.843896 2558 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:02.892658 665 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:43:02.900548 615 consensus_peers.cc:487] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90b8bbd25b8646eb93f739aafb4d00f8. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:43:02.901700 544 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:43:02.901744 776 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:43:02.903214 2221 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:02.904222 2575 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:02.917258 2221 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:02.924196 772 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:02.933744 558 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:02.944571 544 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:02.946977 403 catalog_manager.cc:5495] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:02.972615 499 consensus_peers.cc:487] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f37d457494b64bb48d2f06013a9d26c0. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:43:02.976109 665 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:02.977648 2432 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:02.977902 769 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:02.979661 2432 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:02.998303 2554 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:02.999989 665 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.016865 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:03.018117 544 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.022452 769 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.026854 401 catalog_manager.cc:5495] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:03.037101 772 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:43:03.039803 499 consensus_peers.cc:487] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecf1b55970984ed5ad958ef3c517c6b5. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:43:03.039924 664 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:43:03.047290 2432 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260806 03:43:03.048251 2558 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:03.068326 769 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.070988 2558 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.087469 401 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.087984 664 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.090682 544 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
W20260806 03:43:03.100622 499 consensus_peers.cc:487] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e -> Peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Couldn't send request to peer 1914b652c9fb49b694d71a5f5f5752ee. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20fcbc41710a42cd9ee5dacc5d9577fb. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:43:03.103773 769 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:43:03.105245 2558 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:03.113229 664 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:43:03.115751 2554 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:03.124038 772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:03.124703 772 raft_consensus.cc:595] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:03.130337 544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:03.131150 544 raft_consensus.cc:595] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:03.135506 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:43:03.136250 2554 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.137773 769 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.147393 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:03.148262 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:03.147310 664 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.158537 403 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) added. New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } } }
I20260806 03:43:03.296519 2558 raft_consensus.cc:979] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:03.297152 2554 raft_consensus.cc:1067] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:03.299242 769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57212
I20260806 03:43:03.299791 769 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:03.300130 769 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:03.314466 769 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.320477 769 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:03.322170 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:03.322868 547 raft_consensus.cc:3032] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:03.323184 547 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:03.324273 547 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.325433 547 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:03.330559 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:03.331359 665 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:03.343611 547 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:03.344698 665 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:03.345120 727 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:03.346056 2616 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:03.346809 2616 raft_consensus.cc:686] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:03.348450 2616 consensus_queue.cc:237] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.362437 403 catalog_manager.cc:5495] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:03.396206 2651 ts_tablet_manager.cc:927] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:03.402067 2651 tablet_copy_client.cc:316] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:43:03.406986 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet f37d457494b64bb48d2f06013a9d26c0 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:43:03.407809 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet f37d457494b64bb48d2f06013a9d26c0 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-f37d457494b64bb48d2f06013a9d26c0
I20260806 03:43:03.421783 575 tablet_copy_source_session.cc:205] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:43:03.429481 2651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f37d457494b64bb48d2f06013a9d26c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:03.445078 2616 raft_consensus.cc:979] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:03.445989 2576 raft_consensus.cc:1067] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:03.448078 665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:43:03.448601 665 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:03.448935 665 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:03.460875 665 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.464083 2654 ts_tablet_manager.cc:927] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:03.467581 2651 tablet_copy_client.cc:791] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:43:03.468578 2651 tablet_copy_client.cc:655] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:43:03.469381 665 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:03.471447 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:03.472047 547 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:03.483321 2651 tablet_copy_client.cc:531] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:43:03.484175 547 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:03.493278 614 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:03.495491 2635 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:03.496120 2635 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:03.497313 2635 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.499890 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:03.500777 769 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:03.501157 769 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:03.502182 769 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:03.509168 769 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:03.548285 769 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:03.547973 403 catalog_manager.cc:5495] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 3, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:03.553635 2651 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:43:03.557860 2654 tablet_copy_client.cc:316] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:43:03.571553 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet ecf1b55970984ed5ad958ef3c517c6b5 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:43:03.572108 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet ecf1b55970984ed5ad958ef3c517c6b5 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-ecf1b55970984ed5ad958ef3c517c6b5
I20260806 03:43:03.597412 2661 ts_tablet_manager.cc:927] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:03.597414 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 74667.8us
I20260806 03:43:03.616053 575 tablet_copy_source_session.cc:205] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:43:03.622145 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecf1b55970984ed5ad958ef3c517c6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:03.627627 2664 ts_tablet_manager.cc:927] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Initiating tablet copy from peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:03.633477 2661 tablet_copy_client.cc:316] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.66:41639
I20260806 03:43:03.635174 2664 tablet_copy_client.cc:316] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Beginning tablet copy session from remote peer at address 127.30.138.65:45761
I20260806 03:43:03.641412 691 tablet_copy_service.cc:140] P d8469561a52c4bbc81dce786f1211ee6: Received BeginTabletCopySession request for tablet 90b8bbd25b8646eb93f739aafb4d00f8 from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:46602)
I20260806 03:43:03.641871 575 tablet_copy_service.cc:140] P ada5ef6656cb4420baf77a4b4342909e: Received BeginTabletCopySession request for tablet 20fcbc41710a42cd9ee5dacc5d9577fb from peer 1914b652c9fb49b694d71a5f5f5752ee ({username='slave'} at 127.0.0.1:58044)
I20260806 03:43:03.641948 691 tablet_copy_service.cc:161] P d8469561a52c4bbc81dce786f1211ee6: Beginning new tablet copy session on tablet 90b8bbd25b8646eb93f739aafb4d00f8 from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:46602: session id = 1914b652c9fb49b694d71a5f5f5752ee-90b8bbd25b8646eb93f739aafb4d00f8
I20260806 03:43:03.642560 575 tablet_copy_service.cc:161] P ada5ef6656cb4420baf77a4b4342909e: Beginning new tablet copy session on tablet 20fcbc41710a42cd9ee5dacc5d9577fb from peer 1914b652c9fb49b694d71a5f5f5752ee at {username='slave'} at 127.0.0.1:58044: session id = 1914b652c9fb49b694d71a5f5f5752ee-20fcbc41710a42cd9ee5dacc5d9577fb
I20260806 03:43:03.649580 2651 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:43:03.650655 2651 tablet_bootstrap.cc:492] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:43:03.650669 2654 tablet_copy_client.cc:791] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:43:03.651358 2654 tablet_copy_client.cc:655] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:43:03.651486 2651 ts_tablet_manager.cc:1397] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.098s user 0.016s sys 0.008s
I20260806 03:43:03.656771 2654 tablet_copy_client.cc:531] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:43:03.659689 2651 raft_consensus.cc:348] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.660672 2651 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:43:03.661324 2651 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.666297 575 tablet_copy_source_session.cc:205] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:43:03.668682 2651 ts_tablet_manager.cc:1428] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.017s user 0.005s sys 0.004s
I20260806 03:43:03.656970 691 tablet_copy_source_session.cc:205] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:43:03.670526 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-f37d457494b64bb48d2f06013a9d26c0 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:43:03.671159 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-f37d457494b64bb48d2f06013a9d26c0 on tablet f37d457494b64bb48d2f06013a9d26c0 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:03.671891 2664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fcbc41710a42cd9ee5dacc5d9577fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:03.678753 2661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b8bbd25b8646eb93f739aafb4d00f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:43:03.690516 2654 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:43:03.699210 2664 tablet_copy_client.cc:791] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:43:03.699872 2664 tablet_copy_client.cc:655] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:43:03.703390 2664 tablet_copy_client.cc:531] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:43:03.706465 2661 tablet_copy_client.cc:791] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 0 data blocks...
I20260806 03:43:03.707056 2661 tablet_copy_client.cc:655] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Starting download of 1 WAL segments...
I20260806 03:43:03.710932 2661 tablet_copy_client.cc:531] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:43:03.713982 2664 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:43:03.726256 2654 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:43:03.729020 2654 tablet_bootstrap.cc:492] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:43:03.729761 2654 ts_tablet_manager.cc:1397] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.004s
I20260806 03:43:03.732509 2661 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap starting.
I20260806 03:43:03.732959 2654 raft_consensus.cc:348] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.733916 2654 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:43:03.734588 2654 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.737213 2654 ts_tablet_manager.cc:1428] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260806 03:43:03.742563 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ecf1b55970984ed5ad958ef3c517c6b5 received from {username='slave'} at 127.0.0.1:58044
I20260806 03:43:03.742923 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-ecf1b55970984ed5ad958ef3c517c6b5 on tablet ecf1b55970984ed5ad958ef3c517c6b5 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:03.749882 562 raft_consensus.cc:1261] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:03.751681 2662 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:03.785776 2664 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:43:03.786782 2664 tablet_bootstrap.cc:492] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:43:03.787703 2664 ts_tablet_manager.cc:1397] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.074s user 0.025s sys 0.016s
I20260806 03:43:03.791649 2664 raft_consensus.cc:348] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.792598 2664 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:43:03.793372 2664 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.796018 2664 ts_tablet_manager.cc:1428] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:43:03.797783 575 tablet_copy_service.cc:342] P ada5ef6656cb4420baf77a4b4342909e: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-20fcbc41710a42cd9ee5dacc5d9577fb received from {username='slave'} at 127.0.0.1:58044
I20260806 03:43:03.798147 575 tablet_copy_service.cc:433] P ada5ef6656cb4420baf77a4b4342909e: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-20fcbc41710a42cd9ee5dacc5d9577fb on tablet 20fcbc41710a42cd9ee5dacc5d9577fb with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:03.805608 2661 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:43:03.806561 2661 tablet_bootstrap.cc:492] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Bootstrap complete.
I20260806 03:43:03.807253 2661 ts_tablet_manager.cc:1397] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent bootstrapping tablet: real 0.075s user 0.026s sys 0.005s
I20260806 03:43:03.809861 2661 raft_consensus.cc:348] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.810822 2661 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Initialized, Role: LEARNER
I20260806 03:43:03.811535 2661 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:03.814005 2661 ts_tablet_manager.cc:1428] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260806 03:43:03.815662 691 tablet_copy_service.cc:342] P d8469561a52c4bbc81dce786f1211ee6: Request end of tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-90b8bbd25b8646eb93f739aafb4d00f8 received from {username='slave'} at 127.0.0.1:46602
I20260806 03:43:03.816052 691 tablet_copy_service.cc:433] P d8469561a52c4bbc81dce786f1211ee6: ending tablet copy session 1914b652c9fb49b694d71a5f5f5752ee-90b8bbd25b8646eb93f739aafb4d00f8 on tablet 90b8bbd25b8646eb93f739aafb4d00f8 with peer 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:03.834733 665 raft_consensus.cc:1261] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:03.836294 2517 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:03.867578 1671 raft_consensus.cc:1203] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:43:03.891518 564 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:03.892947 2221 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:03.903913 769 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:03.906500 2635 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:03.923926 1664 raft_consensus.cc:1203] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:43:03.977845 1664 raft_consensus.cc:1203] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:43:04.009168 1664 raft_consensus.cc:1203] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:43:04.210776 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:04.264333 769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:04.264910 769 raft_consensus.cc:595] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:04.268086 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:04.268748 564 raft_consensus.cc:595] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:04.271380 668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:04.271914 668 raft_consensus.cc:595] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:04.273617 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 03:43:04.275302 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:04.275854 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:04.319428 2432 raft_consensus.cc:1050] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:43:04.321651 2432 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.337208 769 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.338141 1670 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.338301 668 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.339634 2554 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:04.340574 2659 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:04.342558 2659 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:04.369261 2683 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.374254 2432 raft_consensus.cc:1050] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:43:04.376536 2432 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.381856 665 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.395341 664 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.398234 776 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.401341 2659 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
W20260806 03:43:04.454478 2685 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20260806 03:43:04.458119 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 69485us
I20260806 03:43:04.462529 1670 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.451058 400 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:04.457866 2695 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260806 03:43:04.486140 1664 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:43:04.576959 776 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.551404 2691 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260806 03:43:04.613049 2554 raft_consensus.cc:1050] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
I20260806 03:43:04.615589 2554 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.630321 2635 raft_consensus.cc:1050] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: attempting to promote NON_VOTER 1914b652c9fb49b694d71a5f5f5752ee to VOTER
W20260806 03:43:04.630828 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 214751us
I20260806 03:43:04.664074 2708 raft_consensus.cc:979] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:04.666177 2708 raft_consensus.cc:1067] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Signalling peer 0374a84d5e9f4e3c8d8287e2382379c8 to start an election
I20260806 03:43:04.676118 2635 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.689433 2432 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.695456 1670 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:04.698163 776 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:04.706231 668 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:04.708425 2695 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 03:43:04.710448 2699 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:04.722554 2695 raft_consensus.cc:979] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:04.723203 2695 raft_consensus.cc:1067] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:04.722493 1664 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.729871 668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:48530
I20260806 03:43:04.729399 2659 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:04.730647 668 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:04.731024 668 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:04.747869 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 50540us
I20260806 03:43:04.754336 564 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.757757 769 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.761785 402 catalog_manager.cc:5495] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:04.759505 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 100798us
I20260806 03:43:04.776150 1671 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:04.737890 668 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.761381 665 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.792850 668 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:04.793776 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:04.794399 1671 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:04.794556 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:04.795094 558 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:04.795388 558 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:04.796319 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 143300us
W20260806 03:43:04.797075 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 99281.1us
I20260806 03:43:04.797865 2554 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:04.802119 1671 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:04.807267 558 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.812573 772 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:04.820858 2659 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:04.868804 615 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:04.865355 772 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:04.935449 558 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:04.966403 558 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:04.972921 776 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:57222
I20260806 03:43:04.973773 776 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:04.974206 776 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:04.947273 2682 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20260806 03:43:04.983840 558 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:04.992367 2718 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 03:43:04.993743 2716 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260806 03:43:05.022289 776 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:05.032445 776 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:05.007920 2718 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:05.130260 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 326466us
I20260806 03:43:05.147240 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:05.148124 1668 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:05.158532 2695 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:05.182564 1668 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:05.186677 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:05.187525 664 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:05.187870 664 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:05.178813 2690 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260806 03:43:05.221400 668 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260806 03:43:05.234450 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 365771us
I20260806 03:43:05.208930 664 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:05.242982 664 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:05.299105 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 57720.6us
I20260806 03:43:05.306691 728 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:05.163775 2718 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:05.325724 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:05.316329 1664 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:43:05.372298 1660 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:05.375666 2723 raft_consensus.cc:979] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:05.384622 1670 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEARNER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:05.414875 664 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:05.427289 2732 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
W20260806 03:43:05.455843 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 223078us
I20260806 03:43:05.464286 2748 raft_consensus.cc:1067] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
W20260806 03:43:05.470906 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 164351us
I20260806 03:43:05.500200 776 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:05.533294 2554 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260806 03:43:05.540872 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 228788us
W20260806 03:43:05.543424 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 44801.8us
I20260806 03:43:05.549346 562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:35214
I20260806 03:43:05.549912 562 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:05.550216 562 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:05.539129 2754 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 03:43:05.629706 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 75935.6us
W20260806 03:43:05.628469 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 60969.2us
W20260806 03:43:05.636699 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 63282us
W20260806 03:43:05.642582 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 95945.2us
W20260806 03:43:05.689254 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 51768.6us
W20260806 03:43:05.895048 2718 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.001s sys 0.005s
W20260806 03:43:06.069613 2718 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.003s sys 0.008s
I20260806 03:43:05.749488 2754 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:06.068776 1660 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:06.070924 2754 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:05.749806 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 75615.3us
I20260806 03:43:06.075505 2774 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.076264 2774 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.082444 2776 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:05.802608 2697 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:06.064754 2724 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:05.731451 2760 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.364s
W20260806 03:43:06.102281 562 consensus_meta.cc:287] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.552s user 0.007s sys 0.005s
I20260806 03:43:06.155241 562 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:06.096019 2695 consensus_meta.cc:287] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.931s user 0.010s sys 0.010s
I20260806 03:43:06.177212 2432 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:06.196372 2432 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.059243 668 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:06.190421 2743 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.202092 1670 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:06.204506 558 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:06.105870 2765 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.090791 2776 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.193186 562 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:06.107367 2699 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.279594 2432 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.155380 2724 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.314172 772 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:06.312845 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 104952us
I20260806 03:43:06.317076 2724 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.249209 2743 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.322050 2743 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:06.323012 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 65954.4us
W20260806 03:43:06.323751 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 130198us
I20260806 03:43:06.295415 2774 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:06.328437 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 87635.5us
I20260806 03:43:06.335538 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:06.336473 1668 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:06.267220 2699 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.353161 2746 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.366397 2699 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.366660 2776 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.375919 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.344540 2783 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:06.373274 2771 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:06.372035 770 raft_consensus.cc:2932] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:06.381287 562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:06.382059 562 raft_consensus.cc:595] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:06.378099 2783 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.383582 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:06.384339 2783 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.364324 2659 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.385645 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:06.365162 2763 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.386435 767 raft_consensus.cc:3032] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:06.386915 767 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:06.388976 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.253520 2765 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:06.400288 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.387893 767 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.401783 767 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:06.406750 563 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:06.358815 2746 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.373533 2760 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:06.411973 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.419968 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.405218 1668 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
W20260806 03:43:06.394991 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 129689us
I20260806 03:43:06.432385 667 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20df53c445204494999245a03bc56a91"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:06.433233 667 raft_consensus.cc:595] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:06.378428 2771 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.405755 2792 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.442971 2792 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.407886 2743 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.438175 2760 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:06.465050 2771 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:06.386222 2659 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:06.484177 2746 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:06.484442 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:06.468122 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:06.499876 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 192) took 1166 ms. Trace:
I20260806 03:43:06.505465 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:06.386950 2763 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.513339 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:06.501726 1658 rpcz_store.cc:276] 0806 03:43:05.326113 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:05.352323 (+ 26210us) service_pool.cc:227] Handling call
0806 03:43:06.102693 (+750370us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:06.159135 (+ 56442us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:06.320999 (+161864us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:06.321033 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:06.321076 (+ 43us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:06.321110 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:06.492555 (+171445us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540044edd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:06.492653 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:06.492867 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":307402880}
I20260806 03:43:06.520810 776 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:06.521461 776 raft_consensus.cc:595] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:06.523524 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 255089us
I20260806 03:43:06.528012 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:06.542910 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 292069us
I20260806 03:43:06.546630 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:06.453019 2781 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260806 03:43:06.551466 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260806 03:43:06.553527 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:06.554564 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:06.562096 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:06.531260 2576 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.519975 2763 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:06.482234 2743 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.589282 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:06.603191 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:06.605620 2700 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.563851 2659 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.607595 1624 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:06.621395 767 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:06.628052 2792 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:06.585896 2576 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.580956 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:06.714927 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 278563us
W20260806 03:43:06.714125 2682 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 03:43:06.786516 2765 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:06.804229 2576 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:06.805279 2760 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:06.715430 2697 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35200 (request call id 1095) took 1944 ms. Trace:
I20260806 03:43:06.843499 2576 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:06.864233 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 316523us
I20260806 03:43:06.841351 2771 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.898465 2709 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:06.636184 2700 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:06.939576 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 223861us
I20260806 03:43:06.917997 2802 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:06.891054 2576 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:06.848661 2697 rpcz_store.cc:276] 0806 03:43:04.751540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:04.751899 (+ 359us) service_pool.cc:227] Handling call
0806 03:43:04.771490 (+ 19591us) log.cc:842] Serialized 501 byte log entry
0806 03:43:06.662328 (+1890838us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1555,"raft.run_cpu_time_us":1043,"raft.run_wall_time_us":1402}
I20260806 03:43:06.974937 2804 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:06.975715 2767 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:06.928035 2738 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:06.907177 2771 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:06.975558 2800 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.320s user 0.000s sys 0.002s
I20260806 03:43:06.969972 2709 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.030351 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.041914 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.042698 2709 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.045158 2725 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.031456 767 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
W20260806 03:43:07.059276 2798 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.268s user 0.002s sys 0.000s
I20260806 03:43:07.059144 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.009577 2576 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.060047 558 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:07.014055 2743 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:07.076666 2803 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 03:43:06.975499 2802 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.002557 2767 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:07.095099 770 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.603s user 0.013s sys 0.003s
I20260806 03:43:07.050479 2697 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.111817 2554 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.112969 499 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:07.121912 2554 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.142856 2554 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.144332 2554 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:07.144965 2554 raft_consensus.cc:686] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:07.047843 2725 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.148102 2709 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.149098 2709 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.157291 2725 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.012010 2804 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.062027 2814 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260806 03:43:07.181797 2554 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.183363 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.185840 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.184340 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.141196 2697 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.191047 2802 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.167418 2804 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.184396 1660 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:07.194893 2700 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.199672 403 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:07.196264 2771 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.189826 1658 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:07.201198 2697 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.179178 2767 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.203016 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.203855 771 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:07.205876 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1523) took 1097 ms. Trace:
I20260806 03:43:07.206362 547 rpcz_store.cc:276] 0806 03:43:06.108585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.204893 (+1096308us) service_pool.cc:227] Handling call
0806 03:43:07.205314 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.205508 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.205540 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.205593 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:07.205634 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.205697 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:07.205811 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:07.201540 403 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:07.190237 2709 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.218962 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.229390 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:07.240712 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.602s user 0.002s sys 0.026s
I20260806 03:43:07.242262 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:07.245690 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.002s sys 0.026s
W20260806 03:43:07.240271 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 272379us
I20260806 03:43:07.248261 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.246500 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 460) took 1770 ms. Trace:
I20260806 03:43:07.249770 1657 rpcz_store.cc:276] 0806 03:43:05.473578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:05.479778 (+ 6200us) service_pool.cc:227] Handling call
0806 03:43:07.223822 (+1744044us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:07.237624 (+ 13802us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.243772 (+ 6148us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.243803 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.243850 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:07.243883 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.243938 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:07.244054 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2791296}
I20260806 03:43:07.251214 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ecf1b55970984ed5ad958ef3c517c6b5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:07.253077 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 393) took 1184 ms. Trace:
I20260806 03:43:07.254716 1663 rpcz_store.cc:276] 0806 03:43:06.067284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:06.067420 (+ 136us) service_pool.cc:227] Handling call
0806 03:43:06.067770 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:06.325777 (+258007us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:06.325826 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:06.325908 (+ 82us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:06.325938 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.251327 (+925389us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540069e954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:07.251378 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:07.251505 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":575053568}
I20260806 03:43:07.256279 2700 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:07.256840 2662 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.253564 2748 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.257383 2662 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.261168 2761 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.264312 2759 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:07.263638 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:07.267613 2707 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.268260 2707 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.270397 2767 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.271173 2767 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.273782 2767 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.275024 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 971) took 1795 ms. Trace:
I20260806 03:43:07.275563 772 rpcz_store.cc:276] 0806 03:43:05.479291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:05.479425 (+ 134us) service_pool.cc:227] Handling call
0806 03:43:05.479734 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:06.314751 (+835017us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.274681 (+959930us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b54002bd4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:07.274917 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1918628736}
I20260806 03:43:07.280656 501 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.306828 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.261901 2761 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:07.349035 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 85480.9us
W20260806 03:43:07.366961 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 100862us
I20260806 03:43:07.367762 2770 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:07.368665 2747 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.369834 2750 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.371362 2432 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:07.267129 2709 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:07.374356 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.377053 501 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.378939 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.262753 2748 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.382148 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.383956 2432 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:07.384335 2432 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.384660 2432 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:07.385394 401 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:07.387086 501 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:07.402804 2768 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:07.407068 2778 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:07.373607 2576 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:07.404124 2746 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:07.370388 2750 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.416829 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.417084 2746 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.421705 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.423214 2821 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.411722 2771 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:07.423820 2821 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.264964 2759 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.429980 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:07.430714 615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.012s sys 0.050s
I20260806 03:43:07.431370 2746 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:07.432688 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.011s sys 0.050s
I20260806 03:43:07.440433 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:07.459828 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.009s sys 0.046s
W20260806 03:43:07.500857 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.012s sys 0.051s
I20260806 03:43:07.381068 2709 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:07.448443 615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.012s sys 0.050s
W20260806 03:43:07.502857 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 938354us
W20260806 03:43:07.501116 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 543270us
I20260806 03:43:07.418653 2576 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.424151 2707 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.421159 2768 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.429093 2662 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:07.447256 1664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.016s sys 0.047s
W20260806 03:43:07.506155 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 65819.2us
I20260806 03:43:07.524446 2752 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:07.506901 618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.019s sys 0.045s
I20260806 03:43:07.507418 2756 raft_consensus.cc:979] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:07.426753 2810 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:07.520714 2432 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:07.525763 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.532004 564 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:07.369215 2747 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.532806 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:07.533295 618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.019s sys 0.045s
I20260806 03:43:07.500607 2754 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.534209 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.535006 2432 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.508188 2576 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:07.503875 2700 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.537882 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.522256 2763 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:07.540838 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.541569 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.540010 1668 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:07.542181 664 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:07.540140 2761 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.426833 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.525952 2746 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:07.530972 1664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s user 0.017s sys 0.047s
I20260806 03:43:07.533100 2814 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.545704 2832 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.525039 2752 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.544394 614 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:07.548714 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.525573 2833 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:07.550055 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.554716 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.541242 2763 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.565671 2221 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:07.567162 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.569577 562 raft_consensus.cc:1226] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260806 03:43:07.569766 2826 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.569965 2779 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.558421 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.561264 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:07.560093 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.030s sys 0.035s
I20260806 03:43:07.614322 2221 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.615042 2221 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:07.612159 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56174.9us
I20260806 03:43:07.547302 2554 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:07.558079 2718 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:07.654879 2718 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.657524 2718 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.541837 2700 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.538751 2754 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.547837 2709 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.571535 2763 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:07.650724 547 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 03:43:07.651315 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.034s sys 0.036s
I20260806 03:43:07.661147 2432 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.661612 2432 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:07.663468 2432 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.573925 2804 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.609939 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.552731 2750 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.609833 2767 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.674010 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:07.560015 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1774) took 1190 ms. Trace:
W20260806 03:43:07.666706 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.012s sys 0.019s
I20260806 03:43:07.572108 2814 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.558648 2746 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:07.698550 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.014s sys 0.019s
W20260806 03:43:07.698881 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 123514us
I20260806 03:43:07.700452 2759 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:07.698979 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 563317us
I20260806 03:43:07.687666 2221 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:07.709031 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 337019us
I20260806 03:43:07.729730 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.704802 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:07.560003 2817 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.735222 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.525918 2768 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.543917 2810 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.733176 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:07.574046 2794 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.687393 2804 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.670120 2709 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.557628 2828 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.741318 768 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:07.744464 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.570453 2826 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.672569 2748 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.753082 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.570571 2779 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.706602 2750 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.680450 2732 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 03:43:07.571873 2832 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.789043 2221 consensus_queue.cc:1046] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:07.771961 775 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:07.776811 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.787431 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:07.777408 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 44398.9us
I20260806 03:43:07.777515 767 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:07.743753 2756 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.730403 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 194) took 2262 ms. Trace:
I20260806 03:43:07.752161 2760 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:07.790357 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.765619 2707 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.703297 2767 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.799650 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.754848 2781 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.808614 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.809389 668 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:07.781908 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.770071 2700 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.568591 2833 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.816094 2755 raft_consensus.cc:3032] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:07.818281 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.822069 768 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:07.830462 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.789551 2827 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:43:07.826220 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 125518us
I20260806 03:43:07.793222 2736 raft_consensus.cc:1067] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:07.826846 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:07.829236 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.835217 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:07.824882 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.788072 2770 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.840581 2221 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.843235 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.801231 2849 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:07.809804 2765 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.848654 2221 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.847014 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.852679 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.854103 674 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:07.855274 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.789467 2750 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.817742 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.862573 2821 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.863983 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.831425 2747 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.780567 2794 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.776247 2709 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.831233 2754 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:07.846292 2755 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:07.890954 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.901274 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.913967 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.915238 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:07.842264 2802 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.900455 2221 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.906857 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:07.877979 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 92705.1us
I20260806 03:43:07.828775 2707 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.838253 2829 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.842132 2801 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.847201 2770 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.932693 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.847731 2718 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260806 03:43:07.936844 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.770226 2817 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:07.948671 562 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:07.936898 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.931382 2754 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:07.947666 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:07.804668 1664 rpcz_store.cc:276] 0806 03:43:05.439927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:05.440726 (+ 799us) service_pool.cc:227] Handling call
0806 03:43:05.508236 (+ 67510us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:05.544631 (+ 36395us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:05.554548 (+ 9917us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:06.624717 (+1070169us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b4c00407140. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32146e2210 00007f32146f2fb2 00007f32146b9ac5 00007f32146e071d 00007f32146e0383 00007f321631a4fb 00007f321631b025 00007f321631c374 00007f321631ba99 00007f3216320ed6 00007f3216315e9b
0806 03:43:06.625933 (+ 1216us) log.cc:842] Serialized 564 byte log entry
0806 03:43:07.564009 (+938076us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:07.569572 (+ 5563us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.577221 (+ 7649us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:07.616544 (+ 39323us) raft_consensus.cc:1671] finished
0806 03:43:07.616572 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:07.695930 (+ 79358us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79300352,"thread_start_us":930290,"threads_started":1,"wal-append.queue_time_us":812966}
I20260806 03:43:07.981259 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.981730 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.799218 2828 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.985436 2554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35200 (request call id 1099) took 2991 ms. Trace:
I20260806 03:43:07.988754 562 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:35214
I20260806 03:43:07.989416 562 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:07.989792 562 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:07.861032 2765 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.922842 2752 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.957623 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:07.994477 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 45075.8us
I20260806 03:43:07.979874 2754 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.896446 2781 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.863023 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.005427 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:07.931195 2814 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:08.018239 2837 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.317s user 0.001s sys 0.001s
I20260806 03:43:08.019898 2779 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:07.985996 2554 rpcz_store.cc:276] 0806 03:43:04.981289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:04.981486 (+ 197us) service_pool.cc:227] Handling call
0806 03:43:05.013809 (+ 32323us) log.cc:842] Serialized 501 byte log entry
0806 03:43:07.972432 (+2958623us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":475,"raft.run_cpu_time_us":1060,"raft.run_wall_time_us":1059}
I20260806 03:43:07.935101 2857 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:07.810186 2756 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:07.861480 2849 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.007231 2746 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.013540 2648 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:08.043222 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 188967us
W20260806 03:43:08.041854 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 118906us
W20260806 03:43:08.033031 2839 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.354s user 0.001s sys 0.001s
I20260806 03:43:08.046132 1624 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:07.990211 2801 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.049015 2749 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.048913 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:08.049669 2749 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.051128 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:08.052322 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.021234 2826 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.028584 2804 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.032840 2857 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:07.937173 2802 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.066496 616 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:08.008937 2794 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.042678 2761 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:07.734557 770 rpcz_store.cc:276] 0806 03:43:06.370061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:06.371324 (+ 1263us) service_pool.cc:227] Handling call
0806 03:43:06.371712 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:06.371909 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:43:07.104623 (+732714us) log.cc:842] Serialized 14 byte log entry
0806 03:43:07.550062 (+445439us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.552244 (+ 2182us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:07.558416 (+ 6172us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:07.559839 (+ 1423us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:07.559955 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2099,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8115,"thread_start_us":421267,"threads_started":1,"wal-append.queue_time_us":455639}
I20260806 03:43:08.086460 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.087136 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.087662 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.072378 562 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.045360 2832 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.095165 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.055766 727 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.090728 2554 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:08.054154 2779 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.097661 663 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:08.063889 2797 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.063313 2732 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.097954 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.101395 562 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:07.937471 2829 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.072660 2857 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:08.089641 2794 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:08.052711 2810 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.141425 2554 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.094470 2767 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.051806 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.102191 2747 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:08.171578 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 125536us
I20260806 03:43:08.188766 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:08.135800 2750 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.142189 2754 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.206126 2845 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.167038 2748 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.100517 2761 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.104776 2779 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.051908 2755 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.068446 2648 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.161522 2797 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.207582 2845 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:08.532349 2855 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.439s user 0.000s sys 0.000s
I20260806 03:43:08.209453 1624 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:08.610857 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 512964us
I20260806 03:43:08.210681 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.744269 2797 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.744889 2869 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.639564 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1465) took 1069 ms. Trace:
I20260806 03:43:08.610713 2828 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.487363 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.193118 2707 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:08.535213 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 354180us
I20260806 03:43:08.613418 2802 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.523073 2709 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:08.713433 1668 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.928s user 0.001s sys 0.009s
W20260806 03:43:08.747095 1668 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s user 0.002s sys 0.009s
I20260806 03:43:08.647324 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 402) took 1278 ms. Trace:
I20260806 03:43:08.312489 2866 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.747908 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.748143 2866 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.749135 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.531621 2756 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.613173 2849 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.629362 2554 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.147797 2768 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.190739 2767 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:08.586211 2826 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:08.149896 2860 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:08.750427 2854 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:08.751564 2860 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.751819 2854 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.752054 2860 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.752259 2854 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.751217 2767 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.747516 1657 rpcz_store.cc:276] 0806 03:43:07.368595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.368740 (+ 145us) service_pool.cc:227] Handling call
0806 03:43:07.369169 (+ 429us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:07.558564 (+189395us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.560580 (+ 2016us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:07.760176 (+199596us) log.cc:842] Serialized 26 byte log entry
0806 03:43:08.010820 (+250644us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:08.024858 (+ 14038us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.646843 (+621985us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54006d24d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.646908 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:08.646966 (+ 58us) raft_consensus.cc:1671] finished
0806 03:43:08.646979 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.647253 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1362,"raft.queue_time_us":9678,"raft.run_cpu_time_us":388,"raft.run_wall_time_us":74440,"spinlock_wait_cycles":376712064,"thread_start_us":205265,"threads_started":1,"wal-append.queue_time_us":308199}
I20260806 03:43:08.750522 2869 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.754969 2768 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.744889 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.638505 2794 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.755587 2768 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.756047 2794 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.473206 2867 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:08.638827 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 144) took 1116 ms. Trace:
I20260806 03:43:08.752718 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.742698 2761 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.751269 2826 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.745599 674 rpcz_store.cc:276] 0806 03:43:07.570039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.570143 (+ 104us) service_pool.cc:227] Handling call
0806 03:43:07.570448 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.854521 (+284073us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.639230 (+784709us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54000b1154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.639429 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":651632384}
I20260806 03:43:08.632185 2817 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.157085 2833 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.313812 2846 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:08.687928 2763 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.028s user 0.013s sys 0.000s
I20260806 03:43:08.750053 2866 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.758785 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.757175 2767 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.757648 2756 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.759614 2826 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.757485 2768 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.757575 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.764539 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.534853 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.765662 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:08.765805 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.207664 2765 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.757509 544 rpcz_store.cc:276] 0806 03:43:07.522256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.522569 (+ 313us) service_pool.cc:227] Handling call
0806 03:43:07.522865 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.812585 (+289720us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.812612 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.812654 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:07.812682 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.638496 (+825814us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b5400086454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.638545 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.638677 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":665847168}
I20260806 03:43:08.766327 2869 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.649569 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.767750 2869 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.767971 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.767901 2860 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.769192 1665 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:08.769392 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.539503 2781 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.667922 2749 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.769982 2756 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.770265 663 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:08.769660 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:08.768265 2869 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.637032 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.771616 664 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:08.771993 727 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.772409 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:08.772466 2869 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.772989 2756 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.773589 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:08.774106 664 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:08.758078 2794 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:08.744395 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 535036us
I20260806 03:43:08.776037 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:08.776667 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.777311 776 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:08.778251 2756 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:08.778954 2756 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:08.779320 2756 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:08.780032 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.780692 766 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:08.781294 2829 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.782593 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.782933 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.783252 773 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:08.783634 766 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:08.747795 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 471) took 1001 ms. Trace:
I20260806 03:43:08.785609 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:08.622956 2576 consensus_meta.cc:287] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.079s user 0.003s sys 0.011s
I20260806 03:43:08.785528 1668 rpcz_store.cc:276] 0806 03:43:07.746540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.747432 (+ 892us) service_pool.cc:227] Handling call
0806 03:43:07.770874 (+ 23442us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:07.771246 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.771274 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:07.780692 (+ 9418us) log.cc:842] Serialized 501 byte log entry
0806 03:43:08.747470 (+966778us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:08.747509 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.747573 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:08.747628 (+ 55us) raft_consensus.cc:1671] finished
0806 03:43:08.747638 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.747744 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":961847,"threads_started":1,"wal-append.queue_time_us":916476}
I20260806 03:43:08.786525 2576 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } }
I20260806 03:43:08.786964 2756 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.788048 772 raft_consensus.cc:1226] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260806 03:43:08.788959 2576 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.749686 2779 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.789247 2756 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.789630 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 985) took 1381 ms. Trace:
I20260806 03:43:08.790006 772 rpcz_store.cc:276] 0806 03:43:07.408385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.411950 (+ 3565us) service_pool.cc:227] Handling call
0806 03:43:07.416352 (+ 4402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:08.788424 (+1372072us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.789319 (+ 895us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400345654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.789572 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3153735424}
I20260806 03:43:08.793136 2770 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.744395 2648 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.763684 2763 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.757328 2867 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:08.804406 1670 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:08.804579 2867 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.805814 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 202) took 1070 ms. Trace:
I20260806 03:43:08.763392 2846 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.795096 2860 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.760658 2845 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.800905 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 882) took 1036 ms. Trace:
I20260806 03:43:08.801831 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1458) took 1254 ms. Trace:
I20260806 03:43:08.809178 558 rpcz_store.cc:276] 0806 03:43:07.764313 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.765039 (+ 726us) service_pool.cc:227] Handling call
0806 03:43:07.800249 (+ 35210us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.822053 (+ 21804us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.822085 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.822122 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:07.822147 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.800675 (+978528us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b540011e4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.800720 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.800845 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49674496}
I20260806 03:43:08.809242 669 rpcz_store.cc:276] 0806 03:43:07.547596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.554260 (+ 6664us) service_pool.cc:227] Handling call
0806 03:43:07.557724 (+ 3464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:08.038595 (+480871us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:08.038626 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:08.038676 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:08.038716 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.801597 (+762881us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b54002aec54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.801646 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.801768 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1097717120}
I20260806 03:43:08.802628 2854 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.806159 1670 rpcz_store.cc:276] 0806 03:43:07.735216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.735386 (+ 170us) service_pool.cc:227] Handling call
0806 03:43:07.743287 (+ 7901us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:08.804803 (+1061516us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.805599 (+ 796us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540044edd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.805757 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2440008960}
I20260806 03:43:08.815292 668 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:08.817175 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.818889 2763 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:08.723856 2857 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.559s user 0.010s sys 0.000s
W20260806 03:43:08.810942 1623 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.000s sys 0.013s
I20260806 03:43:08.821847 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.824105 2801 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:08.825323 1623 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.612s user 0.000s sys 0.014s
I20260806 03:43:08.826884 2846 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.827396 499 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.827910 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:08.828819 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 886) took 1022 ms. Trace:
I20260806 03:43:08.829227 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f37d457494b64bb48d2f06013a9d26c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:08.831054 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.829403 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.833416 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:08.833953 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.829149 548 rpcz_store.cc:276] 0806 03:43:07.806123 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:08.827465 (+1021342us) service_pool.cc:227] Handling call
0806 03:43:08.828756 (+ 1291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:08.835724 2801 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.835906 772 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:08.836591 2849 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.837216 2849 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.839234 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.839874 1670 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:08.841209 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:08.841744 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.841765 772 raft_consensus.cc:3032] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:08.843199 772 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:08.843556 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.846308 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:08.846850 2862 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.706s user 0.004s sys 0.009s
I20260806 03:43:08.844165 773 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:08.848158 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.849452 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.852510 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.855080 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.855717 2801 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:08.856913 2801 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:08.857291 2801 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:08.829917 2857 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.858283 1662 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 03:43:08.858526 2554 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:08.859261 2857 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:08.859061 2554 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.860849 2554 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.855248 2691 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:08.838552 1665 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:08.861091 2771 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.861706 2691 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.863058 1657 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:08.862468 2771 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.870703 2771 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:08.864377 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.867847 772 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.872639 772 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:08.873543 2867 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.858139 2801 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:08.864187 2691 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.880334 728 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:08.885568 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.886552 2768 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:08.887847 2756 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:08.888129 2815 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:08.888288 2756 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.888504 2815 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:08.888720 2756 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.886632 2691 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:08.897677 772 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:08.893378 664 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:08.902493 2768 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.890020 1668 raft_consensus.cc:2932] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:08.902320 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.905136 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.909663 2554 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:08.909852 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.910179 2554 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.914295 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.926424 2554 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.926988 2554 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.928054 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:08.931025 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:08.931784 772 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:08.933014 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:08.938766 1624 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:08.942839 1624 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:08.929617 615 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:08.948024 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:08.948793 2554 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.952082 2763 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:08.959136 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:08.959299 2763 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:08.959985 2763 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.960517 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:08.962556 2869 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:08.963080 2869 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:08.963511 2869 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:08.952962 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:08.953908 501 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:08.969329 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 990) took 1099 ms. Trace:
I20260806 03:43:08.930760 2867 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.924407 2801 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:08.952239 2802 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.955832 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:08.969859 768 rpcz_store.cc:276] 0806 03:43:07.870265 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.901267 (+ 31002us) service_pool.cc:227] Handling call
0806 03:43:07.934229 (+ 32962us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.984772 (+ 50543us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.984862 (+ 90us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.984943 (+ 81us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:07.984992 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:08.969040 (+984048us) spinlock_profiling.cc:243] Waited 50.2 ms on lock 0x7b5400185e54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:08.969106 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:08.969254 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":115469696}
I20260806 03:43:08.988365 2888 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.984349 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:08.907534 2768 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:08.994670 2768 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:08.997856 2809 raft_consensus.cc:979] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:08.998080 2779 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:08.998482 2794 raft_consensus.cc:1067] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:09.000669 2794 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:08.997865 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.011615 769 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:09.029120 2768 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:09.030233 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 480) took 1220 ms. Trace:
I20260806 03:43:09.029824 2768 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.030849 1664 rpcz_store.cc:276] 0806 03:43:07.810006 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:08.197928 (+387922us) service_pool.cc:227] Handling call
0806 03:43:08.508606 (+310678us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:08.749272 (+240666us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:08.749311 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:08.749369 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:08.749401 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:09.029929 (+280528us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54005e3654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:09.029990 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:09.030164 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522161536}
I20260806 03:43:09.032580 2768 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:09.033187 727 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.114s (SENT) [suppressed 3 similar messages]
I20260806 03:43:08.980039 2833 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:08.945438 2876 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:43:09.045581 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 95671.6us
I20260806 03:43:08.945403 2849 consensus_queue.cc:1046] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 03:43:08.982708 2648 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:09.059808 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 963100us
I20260806 03:43:09.066299 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:09.067308 2857 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:09.067852 2857 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:09.068059 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.068389 2857 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:09.069380 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.069571 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.069443 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.070617 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.070907 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.072381 766 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:09.073654 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:09.074564 2873 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:09.075093 2873 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.075541 2873 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.061831 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 475) took 1295 ms. Trace:
I20260806 03:43:09.058773 2888 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:09.079419 2794 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:09.079133 2888 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.079913 2794 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:09.078861 1671 rpcz_store.cc:276] 0806 03:43:07.765998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.767161 (+ 1163us) service_pool.cc:227] Handling call
0806 03:43:07.767431 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.788913 (+ 21482us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.788941 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.788979 (+ 38us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:07.789005 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:09.061516 (+1272511us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b54004f2754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:09.061580 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:09.061756 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48945024}
I20260806 03:43:09.083417 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:09.050800 2854 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:09.087390 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 53451.9us
I20260806 03:43:09.046996 2876 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:09.056825 2648 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:09.094676 2833 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:09.094926 2849 raft_consensus.cc:3032] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:08.985970 2815 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.132529 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.133350 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.096889 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.098239 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.135293 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.130761 2833 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.097378 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.097563 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.097878 2221 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.124086 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.125459 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.080873 2794 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.137251 2221 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.137476 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.137586 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.141503 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.143482 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.144124 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.089354 2768 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260806 03:43:09.144578 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.145342 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.144661 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.145947 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.156509 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.157533 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.159922 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.160844 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.090197 2876 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:09.161830 2221 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.132727 2648 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.162159 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.098762 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.162075 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.166682 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.168171 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.163275 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.149873 2784 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:09.166648 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.172267 671 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:09.167347 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.139374 2725 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.131661 2849 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:09.138402 2883 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:09.142623 2576 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:09.144901 2846 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:09.182003 2221 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:09.178537 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1473) took 1336 ms. Trace:
I20260806 03:43:09.186952 2221 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.186888 665 rpcz_store.cc:276] 0806 03:43:07.842233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.924864 (+ 82631us) service_pool.cc:227] Handling call
0806 03:43:07.925282 (+ 418us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:08.032565 (+107283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:08.032612 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:08.032671 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:08.032738 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:09.178299 (+1145561us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400386b54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:09.178342 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:09.178477 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":246090496}
I20260806 03:43:09.183362 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.158540 2749 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:09.195163 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:09.170087 2815 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:09.145428 2662 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260806 03:43:09.171494 2770 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:09.177492 2794 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.188373 669 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:09.243840 2221 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.184397 2802 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.194357 2771 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:09.253053 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 94114.4us
W20260806 03:43:09.250273 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56388us
I20260806 03:43:09.204661 2691 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:09.205029 2888 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.185174 2725 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:09.211810 2814 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.244515 2746 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:09.249511 2752 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:09.269152 2771 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.289613 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.290198 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.290485 2771 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.283493 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.286510 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.255152 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.291457 544 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:09.291741 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 472) took 1544 ms. Trace:
I20260806 03:43:09.290410 558 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:09.292608 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.291495 562 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:09.280212 2815 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.215575 2770 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.293179 563 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:09.273800 2888 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:09.292243 1663 rpcz_store.cc:276] 0806 03:43:07.746904 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.753436 (+ 6532us) service_pool.cc:227] Handling call
0806 03:43:07.801124 (+ 47688us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:07.897860 (+ 96736us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:07.903029 (+ 5169us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:07.916761 (+ 13732us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:07.923583 (+ 6822us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:09.291438 (+1367855us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b54006414d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:09.291508 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:09.291667 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":132621696}
I20260806 03:43:09.256402 2802 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.271972 2691 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.279587 2814 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.286854 2746 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.296392 2752 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } attrs { replace: true } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.389942 2869 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.437589 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.272461 2849 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.438285 2833 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.316576 2815 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.369839 2883 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.413317 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.372876 2888 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.438580 2784 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:09.464819 2833 consensus_queue.cc:1159] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" responder_term: 2 status { last_received { term: 2 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 2 index: 4 } }
I20260806 03:43:09.450078 2897 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:09.474934 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 173347us
I20260806 03:43:09.476339 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.477741 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.480322 563 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
W20260806 03:43:09.481158 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 397831us
I20260806 03:43:09.482821 501 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:09.459776 2849 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.486690 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.486461 547 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:09.493387 2770 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.494772 2725 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:09.494949 2802 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.498965 616 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:09.500691 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.486138 2897 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.504338 2869 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.457269 2883 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.504654 2752 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.495503 2771 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:09.505522 2771 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:09.505913 2771 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.530995 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:09.532437 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.533147 563 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:09.535209 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.535871 563 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:09.478966 2784 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.537212 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.539556 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.542659 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.504678 2897 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.546591 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.547410 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.554652 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.555071 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.555928 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.557081 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:09.559517 2756 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.653s user 0.009s sys 0.001s
I20260806 03:43:09.559633 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.560060 2756 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.561064 2746 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.554095 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.528929 2802 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:09.524844 2902 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:09.552136 2907 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:09.527635 2883 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.563278 1622 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.592836 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.593129 2902 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.593508 767 raft_consensus.cc:1799] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Handling vote request from an unknown peer d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.596835 767 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 2: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:09.598106 2854 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:09.602547 2815 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.551647 2817 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:09.610277 2763 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.650s user 0.013s sys 0.000s
I20260806 03:43:09.611320 2756 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.613193 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.618808 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.619580 2888 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.620530 1662 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 03:43:09.621018 2784 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:09.603508 2907 raft_consensus.cc:686] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:09.668569 2907 consensus_queue.cc:237] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:09.576747 2869 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:09.591235 2857 consensus_meta.cc:287] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.510s user 0.019s sys 0.010s
I20260806 03:43:09.671211 2857 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.535051 2752 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260806 03:43:09.591585 2814 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.600276 2902 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:09.704998 2554 consensus_queue.cc:1046] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:09.705919 2554 raft_consensus.cc:3032] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:09.706225 2554 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:09.651739 2873 consensus_meta.cc:287] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.576s user 0.011s sys 0.003s
I20260806 03:43:09.582607 2860 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:09.672823 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 126474us
I20260806 03:43:09.715133 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:09.699191 401 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35374 (request call id 272) took 3354 ms. Trace:
I20260806 03:43:09.715765 776 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.577270 2691 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:09.702417 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.702551 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.709540 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:09.709081 664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:09.712049 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.706965 2554 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.842114 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:09.843130 664 raft_consensus.cc:595] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:09.844015 2554 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:09.847707 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:09.848618 1665 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:09.850485 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.850824 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:09.866889 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.715739 2873 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.845798 402 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:09.853149 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:09.891248 776 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:09.716097 401 rpcz_store.cc:276] 0806 03:43:06.319084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:06.319420 (+ 336us) service_pool.cc:227] Handling call
0806 03:43:09.673101 (+3353681us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:07.928905 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:07.929437 (+ 532us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:07.929464 (+ 27us) write_op.cc:435] Acquired schema lock
0806 03:43:07.929486 (+ 22us) tablet.cc:660] Decoding operations
0806 03:43:07.950205 (+ 20719us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:07.983479 (+ 33274us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:07.993735 (+ 10256us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:43:08.026845 (+ 33110us) tablet.cc:699] Row locks acquired
0806 03:43:08.026863 (+ 18us) write_op.cc:260] PREPARE: finished
0806 03:43:08.042437 (+ 15574us) write_op.cc:270] Start()
0806 03:43:08.051937 (+ 9500us) write_op.cc:276] Timestamp: P: 1785987788041160 usec, L: 0
0806 03:43:08.051963 (+ 26us) op_driver.cc:331] REPLICATION: starting
0806 03:43:08.151762 (+ 99799us) log.cc:842] Serialized 536 byte log entry
0806 03:43:09.047596 (+895834us) op_driver.cc:444] REPLICATION: finished
0806 03:43:09.095305 (+ 47709us) write_op.cc:301] APPLY: starting
0806 03:43:09.095404 (+ 99us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:09.095545 (+ 141us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:09.095674 (+ 129us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:09.095923 (+ 249us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:09.096112 (+ 189us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:43:09.096177 (+ 65us) write_op.cc:312] APPLY: finished
0806 03:43:09.096397 (+ 220us) log.cc:842] Serialized 23 byte log entry
0806 03:43:09.128024 (+ 31627us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:09.128125 (+ 101us) write_op.cc:454] Released schema lock
0806 03:43:09.128181 (+ 56us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:09.128515 (+ 334us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:09.128553 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":100752,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":47435,"apply.run_cpu_time_us":2138,"apply.run_wall_time_us":33539,"num_ops":1,"prepare.queue_time_us":525925,"prepare.run_cpu_time_us":19901,"prepare.run_wall_time_us":695749,"replication_time_us":990508,"thread_start_us":9369,"threads_started":1}]]}
W20260806 03:43:09.901842 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 48770.7us
I20260806 03:43:09.662317 2763 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:09.662472 2648 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:09.873544 2794 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.658s user 0.009s sys 0.006s
I20260806 03:43:09.926604 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:09.932799 663 raft_consensus.cc:2396] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:09.871618 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:09.903601 2801 consensus_meta.cc:287] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.916s user 0.012s sys 0.008s
I20260806 03:43:09.936750 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.887712 2905 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:09.893780 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:09.940061 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 227717us
I20260806 03:43:09.904325 2662 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:09.907910 1664 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:09.946790 769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:09.952163 769 raft_consensus.cc:595] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:09.959383 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 242568us
W20260806 03:43:10.008803 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 63770.9us
I20260806 03:43:09.958420 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:09.950054 1658 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.012544 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:10.017186 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.014340 615 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.020017 1663 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.020556 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:10.022540 2781 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.020587 615 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.027273 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:10.028148 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:10.027621 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:10.028252 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:10.006206 2873 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.032405 776 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 2: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:10.034921 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:10.034654 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.000110 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.004350 2648 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.033962 1624 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:10.005407 2763 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:10.044693 2849 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.556s user 0.001s sys 0.009s
I20260806 03:43:10.049022 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:09.965770 2876 consensus_meta.cc:287] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.799s user 0.013s sys 0.003s
I20260806 03:43:10.039994 768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:10.069521 768 raft_consensus.cc:595] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.054265 561 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:10.072360 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.073648 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.076925 2860 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.024173 2857 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:10.057962 615 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.081704 2849 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.081954 562 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:10.055059 2648 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:10.031517 2828 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.068938 558 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:10.032214 2755 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.072295 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.083006 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:10.084043 2771 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.578s user 0.012s sys 0.000s
I20260806 03:43:10.059417 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.093907 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.109361 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:10.222973 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 189864us
I20260806 03:43:10.249811 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.147696 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:10.134014 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 43860.4us
W20260806 03:43:10.151257 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 45191.2us
I20260806 03:43:10.135507 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.113716 403 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35388 (request call id 316) took 3669 ms. Trace:
I20260806 03:43:10.252219 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:10.252400 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 96123.7us
I20260806 03:43:10.253455 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.253722 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.256520 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.257332 776 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:10.256866 402 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:10.257797 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.258358 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.258841 402 catalog_manager.cc:5495] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: config changed from index 3 to 4, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 1 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:10.253543 562 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:10.262962 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.278510 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.270993 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.271636 775 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 03:43:10.265338 773 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:10.261569 498 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.283841 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.283968 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.284687 769 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:10.226169 2919 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260806 03:43:10.291013 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.151191 542 raft_consensus.cc:2396] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:10.292316 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.293613 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.295070 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.252655 2797 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:10.300264 546 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:10.161571 2700 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:10.297655 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.305864 544 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:10.306870 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.308301 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:10.110209 2648 raft_consensus.cc:686] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:10.297914 614 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.297395 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.304762 498 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.299762 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.308979 2554 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:10.151382 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.310660 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.311249 1665 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:10.311864 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.319501 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.101683 2876 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.322592 2911 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:10.323235 1665 raft_consensus.cc:1226] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.323803 2911 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:10.324435 2911 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:10.299921 2809 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260806 03:43:10.325522 2911 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.333025 2911 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.133134 2771 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.261003 2857 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.314930 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.315209 2554 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:10.335043 2554 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.335280 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 419) took 1035 ms. Trace:
I20260806 03:43:10.335896 564 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:10.335724 1664 rpcz_store.cc:276] 0806 03:43:09.299660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:09.300166 (+ 506us) service_pool.cc:227] Handling call
0806 03:43:09.375841 (+ 75675us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:09.956001 (+580160us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:10.334971 (+378970us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b54006d24d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:10.335210 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1195298176}
I20260806 03:43:10.337164 2771 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.316041 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.272043 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.293488 2888 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.339470 558 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:10.300338 2797 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:10.324808 2876 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.341326 2797 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.272047 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 476) took 2495 ms. Trace:
I20260806 03:43:10.343362 2797 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:10.317245 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.294435 2902 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.238694 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.347254 2554 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.348975 564 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "20df53c445204494999245a03bc56a91"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:37194
I20260806 03:43:10.349697 544 raft_consensus.cc:2396] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:10.103153 2833 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.206s
I20260806 03:43:10.351091 2221 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:10.351248 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 53434.1us
I20260806 03:43:10.351938 562 raft_consensus.cc:1226] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.254753 403 rpcz_store.cc:276] 0806 03:43:06.444076 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:06.446584 (+ 2508us) service_pool.cc:227] Handling call
0806 03:43:10.113622 (+3667038us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:07.897124 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:07.908622 (+ 11498us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:07.908675 (+ 53us) write_op.cc:435] Acquired schema lock
0806 03:43:07.908701 (+ 26us) tablet.cc:660] Decoding operations
0806 03:43:07.909242 (+ 541us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:07.909340 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:07.909382 (+ 42us) tablet.cc:683] Acquiring locks for 2 operations
0806 03:43:07.909819 (+ 437us) tablet.cc:699] Row locks acquired
0806 03:43:07.909849 (+ 30us) write_op.cc:260] PREPARE: finished
0806 03:43:07.910040 (+ 191us) write_op.cc:270] Start()
0806 03:43:07.910186 (+ 146us) write_op.cc:276] Timestamp: P: 1785987787909985 usec, L: 0
0806 03:43:07.910216 (+ 30us) op_driver.cc:331] REPLICATION: starting
0806 03:43:07.911191 (+ 975us) log.cc:842] Serialized 982 byte log entry
0806 03:43:08.838454 (+927263us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:08.900131 (+ 61677us) op_driver.cc:444] REPLICATION: finished
0806 03:43:08.998117 (+ 97986us) write_op.cc:301] APPLY: starting
0806 03:43:08.998271 (+ 154us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:09.052060 (+ 53789us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:09.053418 (+ 1358us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:09.093305 (+ 39887us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:09.093551 (+ 246us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 03:43:09.093620 (+ 69us) write_op.cc:312] APPLY: finished
0806 03:43:09.093929 (+ 309us) log.cc:842] Serialized 29 byte log entry
0806 03:43:09.094152 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:09.094253 (+ 101us) write_op.cc:454] Released schema lock
0806 03:43:09.094311 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:09.094704 (+ 393us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:09.094743 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":530520,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":97714,"apply.run_cpu_time_us":2959,"apply.run_wall_time_us":96958,"num_ops":2,"prepare.queue_time_us":621589,"prepare.run_cpu_time_us":4779,"prepare.run_wall_time_us":81297,"raft.queue_time_us":173467,"raft.run_cpu_time_us":61830,"raft.run_wall_time_us":159217,"replication_time_us":989803,"spinlock_wait_cycles":8830208,"thread_start_us":618469,"threads_started":4,"wal-append.queue_time_us":153190,"wal-append.run_cpu_time_us":27955,"wal-append.run_wall_time_us":2055765}]]}
I20260806 03:43:10.352351 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.356600 564 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:10.440165 564 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.358211 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.316697 2700 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.364069 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.356172 547 raft_consensus.cc:1226] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.367071 614 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:10.443907 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 76923.5us
I20260806 03:43:10.444801 614 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.344973 2801 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.368392 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:10.450666 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:10.352512 2896 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260806 03:43:10.369831 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:10.451336 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:10.451910 2896 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:10.452345 2896 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:10.452771 1657 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:10.370929 2648 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.373210 2849 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.374096 2761 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:10.375447 2554 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.377315 2907 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:10.359594 2876 consensus_queue.cc:1046] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260806 03:43:10.381156 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.381726 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.384078 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:10.342797 1658 rpcz_store.cc:276] 0806 03:43:07.776946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.782983 (+ 6037us) service_pool.cc:227] Handling call
0806 03:43:08.996359 (+1213376us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:08.996595 (+ 236us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:10.021246 (+1024651us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:10.271720 (+250474us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b5400550ad4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:10.271951 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2193754752}
I20260806 03:43:10.392345 2923 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:10.396945 2748 raft_consensus.cc:979] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:10.397966 2904 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:10.417968 2221 raft_consensus.cc:3032] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:10.462325 2221 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:10.410804 2925 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.159s user 0.003s sys 0.003s
W20260806 03:43:10.419922 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 105090us
I20260806 03:43:10.427002 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 412) took 2662 ms. Trace:
I20260806 03:43:10.446192 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.463552 2554 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.456415 2794 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.464903 614 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:10.465377 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.447712 2700 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.450875 564 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.451346 674 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:10.453609 2896 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:10.461338 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 79705.5us
I20260806 03:43:10.448943 2718 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.463292 548 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:10.469671 2554 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:10.465937 2221 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.466225 614 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.467743 498 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.473649 2221 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.475656 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.470124 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.470350 564 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.463364 2826 raft_consensus.cc:1067] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Signalling peer d8469561a52c4bbc81dce786f1211ee6 to start an election
I20260806 03:43:10.478235 614 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.480625 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.470474 2797 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:10.467546 2849 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.456231 2761 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.468848 2888 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.482992 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.483880 2746 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:10.466475 661 rpcz_store.cc:276] 0806 03:43:07.763237 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.764889 (+ 1652us) service_pool.cc:227] Handling call
0806 03:43:10.424763 (+2659874us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321b264c6e 00007f321495b807 00007f321495b792 00007f321495b6c1 00007f321495b63c 00007f321495a8f2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:10.425739 (+ 976us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5434635136}
I20260806 03:43:10.467016 2904 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.480298 2718 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.481446 2700 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:10.494257 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.479583 2755 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.462616 2923 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.505309 2849 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.512522 2923 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:10.505313 2801 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.513799 2761 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.480594 2794 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.508459 2700 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:10.508656 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.518471 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:10.519518 2700 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.520035 1658 raft_consensus.cc:1799] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Handling vote request from an unknown peer 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:10.520937 1658 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 03:43:10.521184 659 raft_consensus.cc:2396] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260806 03:43:10.522634 2794 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.484534 2797 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:10.523984 2797 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.509579 2828 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.527978 614 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.526849 2828 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:10.501474 2896 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.501108 2746 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:10.549938 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:10.550797 663 raft_consensus.cc:1799] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Handling vote request from an unknown peer 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:10.551397 663 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 03:43:10.552161 614 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.534900 2779 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.553288 2746 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.556742 2849 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.572299 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.569334 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:10.562516 661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:36554
I20260806 03:43:10.576797 727 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.578766 669 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:10.579311 2779 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.583545 2748 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.599882 2797 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.601687 2797 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.604166 2746 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.604478 2700 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.648586 2746 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:10.726220 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 253678us
I20260806 03:43:10.728523 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.733721 2935 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.736217 501 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.737385 2935 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.755326 2904 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.756126 2911 raft_consensus.cc:3032] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:10.756748 2911 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:10.759424 2911 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.763445 2911 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.769630 2700 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.770632 2700 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:10.771427 2700 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.774869 2700 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:10.823432 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 355576us
I20260806 03:43:10.824770 498 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.825888 2911 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.826697 498 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.827759 2911 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.828599 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:10.830927 498 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.832188 2936 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:10.832756 2936 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:10.833318 2936 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.833401 2935 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.837800 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.839021 2935 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:10.839500 2935 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:10.839972 2935 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:10.847271 402 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.006s sys 0.012s
W20260806 03:43:10.847827 402 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.525s user 0.006s sys 0.012s
I20260806 03:43:10.858444 2927 raft_consensus.cc:3032] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:10.859174 2927 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:10.863543 2927 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.865281 2927 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:10.869432 2935 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.873399 2935 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.892045 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.893159 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.894418 562 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:10.893231 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.897312 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.899822 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.901999 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.904045 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.905222 542 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:10.907200 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.908404 542 raft_consensus.cc:2379] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:10.909729 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.914979 2936 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:10.910807 727 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:10.919176 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.920217 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.921156 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.923308 2896 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:10.923714 2896 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:10.924142 2896 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.928324 2749 raft_consensus.cc:979] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:10.928676 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.928923 2749 raft_consensus.cc:1067] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:10.934489 2936 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:10.933929 548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:10.935436 558 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:10.935542 548 raft_consensus.cc:595] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:10.936388 548 raft_consensus.cc:619] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Rejecting request to transfer leadership because tablet server 0374a84d5e9f4e3c8d8287e2382379c8 is not a voter in the active config
I20260806 03:43:10.939613 727 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.921201 542 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
W20260806 03:43:10.945093 659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1468) took 3206 ms (client timeout 1913 ms). Trace:
W20260806 03:43:10.947870 659 rpcz_store.cc:269] 0806 03:43:07.738785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:07.738906 (+ 121us) service_pool.cc:227] Handling call
0806 03:43:10.944604 (+3205698us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:10.944837 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5793132928}
I20260806 03:43:10.949616 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:43:10.950529 2749 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.951988 674 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:10.946557 2896 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:10.955350 664 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:10.956619 727 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.957469 2896 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:10.958196 2896 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.958626 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.959620 2749 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:10.960484 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.964054 2943 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.966744 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.967792 2896 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.968420 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.969364 2896 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.969486 2749 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.970455 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.971557 2749 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.972723 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.973793 2749 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:10.976665 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.977001 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.977679 542 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:10.982949 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 899) took 1023 ms. Trace:
I20260806 03:43:10.983299 547 rpcz_store.cc:276] 0806 03:43:09.959465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:10.343665 (+384200us) service_pool.cc:227] Handling call
0806 03:43:10.343895 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:10.443101 (+ 99206us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:10.982512 (+539411us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b540011f8d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:10.982720 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27888512}
I20260806 03:43:10.985805 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.986635 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.986755 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.985834 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:10.987376 546 raft_consensus.cc:2362] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2. Re-sending same reply.
I20260806 03:43:10.985870 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.988318 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.988662 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.988798 1622 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.989307 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:10.989733 548 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:10.990635 2923 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.991108 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.988597 542 raft_consensus.cc:2362] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2. Re-sending same reply.
I20260806 03:43:10.991348 2923 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:10.995119 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:10.995621 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:10.996701 1658 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:10.996798 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1580) took 1145 ms. Trace:
I20260806 03:43:10.997102 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:10.998090 2918 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.000152 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.001236 1658 raft_consensus.cc:2379] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:11.001150 2784 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:10.997318 563 rpcz_store.cc:276] 0806 03:43:09.851374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:10.320434 (+469060us) service_pool.cc:227] Handling call
0806 03:43:10.503372 (+182938us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:10.503641 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:10.503677 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:10.503734 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:10.503776 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:10.503846 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:10.996455 (+492609us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54002b0550. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:10.996621 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":419970304}
I20260806 03:43:11.001981 2784 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.008718 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.009639 2923 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.011327 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.012168 2784 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:11.012784 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.012877 1622 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.013013 2872 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.013953 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.014407 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.014750 2923 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:10.585237 661 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:11.015244 2923 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:11.015709 2923 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.017798 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.018435 2896 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.019136 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.019208 2896 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.021571 773 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.025792 1622 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.027310 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:11.027894 2872 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.029134 2872 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.030251 2872 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.038092 2923 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.039359 770 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.041420 2923 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.033897 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.043532 2923 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.044081 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.044124 2923 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.045675 2923 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.047129 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.015756 661 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.055321 2918 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.056445 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:11.057065 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.057713 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:11.057953 547 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:35214
I20260806 03:43:11.058068 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.058734 547 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:11.059092 547 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.057732 563 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.068979 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.077329 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.058076 727 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:11.081987 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.083149 563 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:11.085935 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:11.087184 2836 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.082307 547 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.091235 728 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.088100 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:11.093454 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 512974us
I20260806 03:43:11.095502 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.067229 2784 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:11.096073 2836 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:11.100526 1622 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.101709 2918 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.102329 2918 raft_consensus.cc:686] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.103699 2918 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.107602 2943 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.108316 766 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:11.110136 2857 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:11.110591 562 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 4. Ops: []
I20260806 03:43:11.114430 562 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260806 03:43:11.114811 562 pending_rounds.cc:76] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 4
I20260806 03:43:11.115063 562 pending_rounds.cc:97] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260806 03:43:11.115396 562 raft_consensus.cc:2866] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:11.116393 661 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.117640 2794 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:11.118574 2896 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:11.125607 773 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:11.119159 2749 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.119714 728 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.132856 2896 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.127177 547 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.126536 2794 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:11.097538 2836 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.136432 2896 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.136175 2794 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.131042 766 raft_consensus.cc:1226] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:11.165525 501 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:11.140862 2748 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.168735 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.169597 773 raft_consensus.cc:3032] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:11.142021 2794 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.170244 773 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:11.141676 2857 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 03:43:11.181406 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.182320 1664 raft_consensus.cc:2379] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:11.182631 661 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.179937 2936 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.192157 2936 raft_consensus.cc:686] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:11.193708 402 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35370 (request call id 317) took 2375 ms. Trace:
I20260806 03:43:11.193328 2936 consensus_queue.cc:237] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.197840 755 tablet_service.cc:1514] Processing DeleteTablet for tablet f37d457494b64bb48d2f06013a9d26c0 with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47796
I20260806 03:43:11.179144 2923 consensus_queue.cc:1046] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20260806 03:43:11.206310 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 86446.2us
I20260806 03:43:11.210243 1664 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.215649 548 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.219624 2718 consensus_queue.cc:1046] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:11.220942 2718 raft_consensus.cc:3032] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.202673 773 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.225975 773 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.232355 2707 consensus_queue.cc:1046] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:11.232996 547 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.233384 2707 consensus_queue.cc:1225] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:43:11.233989 2707 raft_consensus.cc:3032] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.234335 2707 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:11.235489 2707 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.236991 2707 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.237711 401 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51378 (request call id 176) took 1161 ms. Trace:
I20260806 03:43:11.237705 766 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:11.238145 401 rpcz_store.cc:276] 0806 03:43:10.075864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:10.077924 (+ 2060us) service_pool.cc:227] Handling call
0806 03:43:11.237501 (+1159577us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97294,"reported_tablets":4}
I20260806 03:43:11.221304 2718 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:11.247678 773 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:11.225490 2749 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.250560 501 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:11.250869 2648 raft_consensus.cc:3032] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:11.243587 2691 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:11.193971 402 rpcz_store.cc:276] 0806 03:43:08.818427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:08.818590 (+ 163us) service_pool.cc:227] Handling call
0806 03:43:10.848272 (+2029682us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:10.848353 (+ 81us) write_op.cc:454] Released schema lock
0806 03:43:11.193600 (+345247us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:10.401273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:10.402406 (+ 1133us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:10.402490 (+ 84us) write_op.cc:435] Acquired schema lock
0806 03:43:10.402547 (+ 57us) tablet.cc:660] Decoding operations
0806 03:43:10.403600 (+ 1053us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:10.403704 (+ 104us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:10.403766 (+ 62us) tablet.cc:683] Acquiring locks for 3 operations
0806 03:43:10.404347 (+ 581us) tablet.cc:699] Row locks acquired
0806 03:43:10.404408 (+ 61us) write_op.cc:260] PREPARE: finished
0806 03:43:10.404601 (+ 193us) write_op.cc:270] Start()
0806 03:43:10.404748 (+ 147us) write_op.cc:276] Timestamp: P: 1785987790404539 usec, L: 0
0806 03:43:10.404779 (+ 31us) op_driver.cc:331] REPLICATION: starting
0806 03:43:10.408460 (+ 3681us) log.cc:842] Serialized 1302 byte log entry
0806 03:43:10.422710 (+ 14250us) op_driver.cc:444] REPLICATION: finished
0806 03:43:10.485303 (+ 62593us) write_op.cc:301] APPLY: starting
0806 03:43:10.489471 (+ 4168us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:10.489635 (+ 164us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:10.492070 (+ 2435us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:10.516741 (+ 24671us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:10.516944 (+ 203us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 03:43:10.517059 (+ 115us) write_op.cc:312] APPLY: finished
0806 03:43:10.517410 (+ 351us) log.cc:842] Serialized 35 byte log entry
0806 03:43:10.517672 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:10.517844 (+ 172us) write_op.cc:454] Released schema lock
0806 03:43:10.517913 (+ 69us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":462948,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":59100,"apply.run_cpu_time_us":3400,"apply.run_wall_time_us":34431,"num_ops":3,"prepare.queue_time_us":78803,"prepare.run_cpu_time_us":12249,"prepare.run_wall_time_us":16096,"raft.queue_time_us":1827,"raft.run_cpu_time_us":12046,"raft.run_wall_time_us":35843,"replication_time_us":17805,"spinlock_wait_cycles":114944,"thread_start_us":575538,"threads_started":4,"wal-append.queue_time_us":8735}]]}
W20260806 03:43:11.232419 2761 consensus_queue.cc:1159] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ada5ef6656cb4420baf77a4b4342909e" responder_term: 4 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:43:11.251235 2648 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.259814 2648 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.260872 2927 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:11.261541 2648 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.267235 2781 raft_consensus.cc:979] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: : Instructing follower ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:11.267860 2781 raft_consensus.cc:1067] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Signalling peer ada5ef6656cb4420baf77a4b4342909e to start an election
I20260806 03:43:11.280704 2718 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.282243 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.281253 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.283232 2718 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.283805 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.283054 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.292855 664 raft_consensus.cc:2396] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:11.261561 2927 raft_consensus.cc:686] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:11.305882 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.313376 2927 consensus_queue.cc:237] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.429647 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.420718 2746 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:11.433477 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:11.433591 2746 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.443684 2554 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:11.443670 2755 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.443958 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.444653 2554 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.446705 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.448568 2746 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.446318 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.447446 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.451102 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:11.451812 2956 tablet_replica.cc:330] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:43:11.462579 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.465269 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.467971 546 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:11.465256 1664 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.466697 1657 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:11.473428 674 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:11.478829 2554 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.478847 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.479816 2849 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:11.480237 1665 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:11.480688 2849 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.489681 2849 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.491132 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.491600 1664 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:11.493495 401 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 2 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:11.495869 401 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 3, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 3 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:11.494642 615 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:11.497110 2965 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 03:43:11.504117 2923 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:11.504725 674 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:11.507264 2849 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:11.513916 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 53548us
I20260806 03:43:11.492221 661 raft_consensus.cc:3032] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.520881 661 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:11.521865 661 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.522966 661 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.523804 1664 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:11.525311 2936 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:11.527388 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:11.527827 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 433470us
I20260806 03:43:11.528321 766 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:11.529327 614 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.530122 2907 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:11.530782 2907 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.518867 615 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:11.535283 2833 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:11.535701 2833 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:11.536064 2833 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.511924 2923 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:11.536700 2923 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.538748 2828 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:11.539323 2828 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:11.539759 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.540516 2828 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.515165 2849 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:11.543917 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.554804 2849 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.571364 2707 consensus_queue.cc:1046] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:11.572506 2707 raft_consensus.cc:3032] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.572847 2707 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:11.575534 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.580955 768 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:11.570003 766 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:11.583388 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.585151 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.585872 669 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.573751 2707 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.595841 659 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:11.599501 2707 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.600530 1657 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:11.605217 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.605897 2956 raft_consensus.cc:2229] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:11.608163 2956 raft_consensus.cc:2258] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:11.608089 2749 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:11.608908 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.609623 663 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:11.617269 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.615415 2956 ts_tablet_manager.cc:1905] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:43:11.618191 766 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.618736 766 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:11.619673 766 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.620384 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.620893 766 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.621137 663 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:11.617614 1657 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:11.622618 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46012.5us
I20260806 03:43:11.623488 498 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.632401 674 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:11.607393 2907 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.641211 2648 consensus_queue.cc:1046] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:11.643319 498 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.602221 2923 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.652122 2809 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:11.654182 2809 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.655699 2974 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:11.655987 2923 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:11.656667 2809 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.656240 2974 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.657161 2911 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.656973 661 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:11.630221 2872 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 03:43:11.666810 498 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.651801 2927 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.669878 2953 raft_consensus.cc:3032] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:11.673228 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.651029 2718 consensus_queue.cc:1046] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260806 03:43:11.674566 772 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:11.677843 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:11.669625 446 debug-util.cc:398] Leaking SignalData structure 0x7b08000550c0 after lost signal to thread 31279
I20260806 03:43:11.653115 2907 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:11.690685 546 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.691079 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.693271 768 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.694625 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.693986 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.691401 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.700014 563 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:11.692853 2809 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:11.701453 2809 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:11.703888 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 59704.3us
I20260806 03:43:11.683465 2953 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.706720 2833 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:11.707753 2953 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.709267 2953 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.711238 1657 raft_consensus.cc:2379] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:11.704931 446 debug-util.cc:398] Leaking SignalData structure 0x7b08000851a0 after lost signal to thread 450
I20260806 03:43:11.713435 2974 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:11.714077 446 debug-util.cc:398] Leaking SignalData structure 0x7b080038c4e0 after lost signal to thread 594
W20260806 03:43:11.715516 446 debug-util.cc:398] Leaking SignalData structure 0x7b080038c000 after lost signal to thread 711
I20260806 03:43:11.705327 2907 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.725864 2836 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:11.733737 563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:37194
I20260806 03:43:11.732672 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:11.734404 563 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:43:11.684547 669 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:11.710187 2833 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:11.729053 2927 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.735589 2836 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.738793 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.739877 770 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:11.740653 2836 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.741858 2911 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:11.743726 563 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:11.744701 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 50141.8us
I20260806 03:43:11.745057 2927 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.745517 498 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.747072 498 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.747179 2849 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.746016 1665 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:11.749442 2691 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.750430 2691 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.751099 2221 raft_consensus.cc:3032] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.751554 2221 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:11.752027 648 tablet_service.cc:1514] Processing DeleteTablet for tablet ecf1b55970984ed5ad958ef3c517c6b5 with delete_type TABLET_DATA_TOMBSTONED (TS d8469561a52c4bbc81dce786f1211ee6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36546
I20260806 03:43:11.752436 2221 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.756496 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:11.756883 2907 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.757424 2974 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:11.757879 2974 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:11.758330 2974 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.695001 2936 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.764266 2221 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.765857 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.766695 1662 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:11.767974 2809 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.769888 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.770617 558 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:11.771078 2849 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:11.771018 728 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:11.774427 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.775712 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:11.778995 446 debug-util.cc:398] Leaking SignalData structure 0x7b0800337a00 after lost signal to thread 811
W20260806 03:43:11.780340 446 debug-util.cc:398] Leaking SignalData structure 0x7b080033b180 after lost signal to thread 1712
I20260806 03:43:11.773703 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.735700 2986 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260806 03:43:11.774521 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:11.786871 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 82374.7us
I20260806 03:43:11.788563 563 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.796159 547 raft_consensus.cc:1226] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:11.803349 563 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.805168 669 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.805886 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:11.806537 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 59375.6us
I20260806 03:43:11.806643 668 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:11.807345 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.807842 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.794313 2836 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:11.808609 663 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:11.808889 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.809011 2836 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:11.809624 2836 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.808807 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.810987 664 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:11.796393 2707 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:11.812291 766 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:11.812364 2902 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.813462 2936 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.810176 498 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.815923 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:11.816264 2691 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:11.828711 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:11.829295 563 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.824841 2986 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:11.831629 2691 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:11.831805 2986 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.833988 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.835194 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.835855 2927 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.836625 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.837255 659 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 4. Ops: []
I20260806 03:43:11.838256 773 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:11.839857 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.839974 2849 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.842885 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:11.843658 1657 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.844129 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.835311 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:11.837301 772 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.836679 614 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:11.849439 376 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 20fcbc41710a42cd9ee5dacc5d9577fb with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:43:11.856568 2554 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:11.833264 2691 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.853715 674 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:11.865792 2554 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:11.866282 2554 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.867542 2221 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:11.868054 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.854642 2993 tablet_replica.cc:330] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: stopping tablet replica
I20260806 03:43:11.874845 2761 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:11.849840 2911 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:11.812780 2707 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.896593 2761 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:11.899669 2761 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:11.902506 2974 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.854300 2987 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260806 03:43:11.905690 2857 raft_consensus.cc:3032] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:11.906304 2857 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:11.906729 2993 raft_consensus.cc:2229] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:11.908253 2993 raft_consensus.cc:2258] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:11.907639 2857 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.863445 2956 ts_tablet_manager.cc:1918] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:43:11.909726 2857 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:11.910403 2956 log.cc:1196] T f37d457494b64bb48d2f06013a9d26c0 P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/f37d457494b64bb48d2f06013a9d26c0
I20260806 03:43:11.911556 2974 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.912005 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:11.914438 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet f37d457494b64bb48d2f06013a9d26c0 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:43:11.874799 2849 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:11.916064 769 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 3
W20260806 03:43:11.896203 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 44087.4us
I20260806 03:43:11.918522 1664 raft_consensus.cc:1226] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:11.922297 563 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:11.935042 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 84176.2us
W20260806 03:43:11.935695 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 92081.5us
I20260806 03:43:11.936702 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.888782 548 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:11.826102 2907 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260806 03:43:11.944873 2554 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:11.944687 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 69485.5us
I20260806 03:43:11.945536 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.946483 2554 consensus_queue.cc:1046] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:11.890100 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.948858 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:11.950672 772 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:11.932190 2849 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:11.957396 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:11.959249 2554 raft_consensus.cc:3032] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:11.959765 2554 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:11.964793 498 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:11.931471 2986 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:11.960695 2554 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:11.969776 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:11.946952 2936 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:11.970913 775 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:11.985342 674 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.959139 2849 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:11.994899 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:11.996672 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 125932us
I20260806 03:43:11.996840 2554 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:11.996687 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:11.995967 2872 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:12.000144 659 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:11.984148 2936 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:12.005046 3001 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:12.006639 2995 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:12.010303 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:12.004058 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:12.033180 664 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:12.030275 769 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:12.010479 614 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:12.034158 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:11.998087 2927 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:12.034905 614 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:12.011284 2872 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:12.041946 2221 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:12.037452 668 raft_consensus.cc:2379] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:12.045398 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 52449.6us
I20260806 03:43:12.031373 2936 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.031149 3001 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:12.050149 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 85457.5us
I20260806 03:43:12.043548 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:12.051648 2993 ts_tablet_manager.cc:1905] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:43:12.034905 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:12.053061 669 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:12.055305 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:12.037041 2809 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:12.046185 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:12.040257 2927 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:12.045310 2872 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:12.108011 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 61924.5us
I20260806 03:43:12.048883 2974 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:12.108681 2221 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:12.112156 2221 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:12.112460 674 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.113200 2221 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.114121 3017 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:12.056774 3001 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:12.107391 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 79020us
I20260806 03:43:12.126103 501 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:12.108523 2927 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:12.116117 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 64974.5us
I20260806 03:43:12.132768 1665 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.135578 664 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:12.115955 2707 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:12.118352 3003 raft_consensus.cc:2741] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 03:43:12.129528 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:12.066236 2718 consensus_queue.cc:1046] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
W20260806 03:43:12.150848 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 95611.8us
I20260806 03:43:12.151542 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:12.121101 2974 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:12.157318 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:12.158241 2554 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:12.161162 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:12.161782 674 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:12.167387 3018 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:12.169760 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:12.170920 659 raft_consensus.cc:2379] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:12.172521 2554 raft_consensus.cc:686] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:12.152832 2995 raft_consensus.cc:686] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:12.191020 2554 consensus_queue.cc:237] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.147020 2923 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 03:43:12.192646 665 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:12.160207 2707 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:12.194456 668 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 03:43:12.202034 3002 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20260806 03:43:12.154803 2828 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20260806 03:43:12.228359 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 55737us
I20260806 03:43:12.193735 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:12.227937 2707 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:12.285056 546 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.286532 2707 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.287468 561 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.289021 564 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.290629 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:12.291839 2836 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.293097 766 raft_consensus.cc:2354] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:12.293910 2836 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.294633 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:12.295252 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 62504.6us
I20260806 03:43:12.313117 2998 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:12.317656 1657 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:12.319618 769 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:12.319449 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:12.321208 668 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.321376 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:12.313731 2998 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.283033 661 raft_consensus.cc:3032] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:12.346436 547 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.362278 562 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:12.368136 2998 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.369491 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:12.369877 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:12.370132 1657 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.339238 661 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:12.280875 2974 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.390990 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:12.401750 3012 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260806 03:43:12.402892 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:12.405652 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:12.410676 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:12.411309 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 262451us
I20260806 03:43:12.391479 2988 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:12.422200 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:12.340668 2995 consensus_queue.cc:237] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.404804 2749 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 03:43:12.440903 2849 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.424906 2761 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:12.448053 3026 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:12.448511 3026 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:12.448863 3026 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:12.407004 2987 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260806 03:43:12.433420 661 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.444756 2748 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:12.452531 2993 ts_tablet_manager.cc:1918] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260806 03:43:12.447997 2857 consensus_meta.cc:287] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.538s user 0.015s sys 0.000s
I20260806 03:43:12.453153 2993 log.cc:1196] T ecf1b55970984ed5ad958ef3c517c6b5 P d8469561a52c4bbc81dce786f1211ee6: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-1-root/wals/ecf1b55970984ed5ad958ef3c517c6b5
I20260806 03:43:12.453670 1664 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:12.446679 3029 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260806 03:43:12.447058 2872 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.447474 3030 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:12.463006 775 raft_consensus.cc:1226] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:12.426200 2907 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
W20260806 03:43:12.503299 3042 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260806 03:43:12.498134 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 109414us
I20260806 03:43:12.503439 2849 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.528769 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:12.529560 775 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.532774 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:12.544747 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:12.545444 547 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.540584 2923 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:12.594308 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:12.594513 668 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:12.604951 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:12.619557 669 raft_consensus.cc:1226] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:12.621920 401 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35388 (request call id 317) took 1310 ms. Trace:
I20260806 03:43:12.533452 376 catalog_manager.cc:4841] TS d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): tablet ecf1b55970984ed5ad958ef3c517c6b5 (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
W20260806 03:43:12.625633 376 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 92198.5us
I20260806 03:43:12.627417 3030 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:12.628528 3030 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.622491 401 rpcz_store.cc:276] 0806 03:43:11.420391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:11.423723 (+ 3332us) service_pool.cc:227] Handling call
0806 03:43:12.621841 (+1198118us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:11.610877 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:11.613286 (+ 2409us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:11.613344 (+ 58us) write_op.cc:435] Acquired schema lock
0806 03:43:11.613363 (+ 19us) tablet.cc:660] Decoding operations
0806 03:43:11.613867 (+ 504us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:11.613989 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:11.614044 (+ 55us) tablet.cc:683] Acquiring locks for 2 operations
0806 03:43:11.614531 (+ 487us) tablet.cc:699] Row locks acquired
0806 03:43:11.614569 (+ 38us) write_op.cc:260] PREPARE: finished
0806 03:43:11.614747 (+ 178us) write_op.cc:270] Start()
0806 03:43:11.614899 (+ 152us) write_op.cc:276] Timestamp: P: 1785987791614679 usec, L: 0
0806 03:43:11.614937 (+ 38us) op_driver.cc:331] REPLICATION: starting
0806 03:43:11.640094 (+ 25157us) log.cc:842] Serialized 919 byte log entry
0806 03:43:11.982410 (+342316us) op_driver.cc:444] REPLICATION: finished
0806 03:43:12.283476 (+301066us) write_op.cc:301] APPLY: starting
0806 03:43:12.323712 (+ 40236us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:12.336070 (+ 12358us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:12.336234 (+ 164us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:12.343486 (+ 7252us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:12.343748 (+ 262us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 03:43:12.343791 (+ 43us) write_op.cc:312] APPLY: finished
0806 03:43:12.349166 (+ 5375us) log.cc:842] Serialized 29 byte log entry
0806 03:43:12.375641 (+ 26475us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:12.382557 (+ 6916us) write_op.cc:454] Released schema lock
0806 03:43:12.390748 (+ 8191us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:12.391073 (+ 325us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:12.391102 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":230276,"apply.run_cpu_time_us":3345,"apply.run_wall_time_us":140835,"num_ops":2,"prepare.queue_time_us":102711,"prepare.run_cpu_time_us":13599,"prepare.run_wall_time_us":107301,"raft.queue_time_us":205421,"raft.run_cpu_time_us":25795,"raft.run_wall_time_us":245154,"replication_time_us":351722,"thread_start_us":322078,"threads_started":4,"wal-append.queue_time_us":57266}]]}
I20260806 03:43:12.640121 661 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:12.652918 3004 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:12.674548 3004 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.677043 3004 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.634059 2761 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:12.679946 2761 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:12.680691 2761 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:12.683856 3050 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260806 03:43:12.599273 2923 raft_consensus.cc:686] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:12.686729 2923 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.701933 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:12.701177 775 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:12.659085 2927 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.506s user 0.009s sys 0.003s
I20260806 03:43:12.703485 1658 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:12.703683 2927 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.608225 2911 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:12.705322 562 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:12.705470 2911 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:12.581257 2872 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:12.706324 2911 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:12.671367 3052 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.063s user 0.000s sys 0.002s
I20260806 03:43:12.672143 768 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:12.699455 2761 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:12.653990 3001 consensus_meta.cc:287] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.514s user 0.006s sys 0.006s
I20260806 03:43:12.715269 499 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:12.714280 2746 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:12.739305 3017 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:12.740046 3017 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:12.740711 3017 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:12.731527 562 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:12.745579 547 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:12.746970 614 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:12.750463 2849 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:12.753634 2833 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:12.758929 2828 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:12.771916 2828 raft_consensus.cc:686] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:12.760299 616 leader_election.cc:398] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:12.774665 616 leader_election.cc:401] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:12.776345 2833 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.780908 2221 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
W20260806 03:43:12.781271 2936 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.724s user 0.006s sys 0.006s
I20260806 03:43:12.782353 402 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35374 (request call id 273) took 1501 ms. Trace:
I20260806 03:43:12.784125 402 rpcz_store.cc:276] 0806 03:43:11.280687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:11.306275 (+ 25588us) service_pool.cc:227] Handling call
0806 03:43:12.782288 (+1476013us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3}
I20260806 03:43:12.806638 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:12.807727 1664 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:12.809970 499 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:12.825076 403 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:12.828608 775 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:12.828250 403 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:12.809084 3030 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.832382 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:12.839445 1664 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.766729 3001 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.836953 403 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 2 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:12.781957 2936 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.846778 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:12.847611 775 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:12.820148 659 raft_consensus.cc:2932] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:12.774037 2828 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.859417 3031 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:12.879035 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:12.854269 1657 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:12.861775 2927 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:12.865556 2949 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:12.871721 3026 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.893214 2949 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.895210 2833 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.882978 1658 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:12.863330 3031 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:12.896756 1662 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:12.896963 3031 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:12.875216 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:12.898550 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:12.860586 3005 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260806 03:43:12.901846 728 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:12.877781 1664 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:12.929394 769 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:12.883940 661 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:12.944110 3018 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260806 03:43:12.938035 1658 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:12.904294 3026 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.954702 2700 consensus_queue.cc:1046] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:12.903937 2707 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:12.956820 2707 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.957988 3041 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:12.960492 3041 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:12.964138 2927 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:12.965337 2700 raft_consensus.cc:3032] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:12.964969 2927 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.965831 2700 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:12.950670 3025 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:12.962072 3041 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.967695 2700 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:12.970690 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:12.966991 3025 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:12.953356 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:12.952512 2872 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 03:43:12.979609 2927 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:12.996291 2700 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:12.972393 2707 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:12.995684 3004 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.020392 2949 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.010541 3025 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:13.016355 768 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:13.023773 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.025458 1663 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:13.029049 3025 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.030133 3025 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.034327 3025 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:13.018151 2911 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.019925 3004 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.007050 3001 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:13.040161 501 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:13.040692 3004 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.041775 2927 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:13.042485 2927 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.043125 2927 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:13.045393 2936 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.046649 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:13.047698 770 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:13.052546 3017 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.054453 547 raft_consensus.cc:2354] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3.
I20260806 03:43:13.063267 728 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:13.067997 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:13.081054 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 142196us
W20260806 03:43:13.083922 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.014s sys 0.019s
W20260806 03:43:13.084825 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.551s user 0.015s sys 0.019s
W20260806 03:43:13.087044 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.005s sys 0.028s
I20260806 03:43:13.088637 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:13.087452 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.006s sys 0.028s
I20260806 03:43:13.073024 3001 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:13.095631 614 leader_election.cc:398] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3.
I20260806 03:43:13.118338 614 leader_election.cc:401] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:13.122258 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:43:13.097630 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 675520us
I20260806 03:43:13.206426 674 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:13.086879 3054 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:13.213238 671 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.016s sys 0.033s
W20260806 03:43:13.213675 671 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s user 0.017s sys 0.033s
W20260806 03:43:13.095679 2995 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.016s sys 0.002s
I20260806 03:43:13.119055 3001 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:13.095839 3018 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:13.238667 3035 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.087162 2849 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.153s
I20260806 03:43:13.239216 3035 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.241848 3035 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.242630 2221 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:13.242745 3058 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:43:13.243117 3015 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.243338 2221 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.202416 3029 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.250097 2221 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.210409 2648 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:13.244648 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 157457us
I20260806 03:43:13.250133 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.252715 661 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.253224 2986 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:13.242555 3055 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.257115 3055 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.138600 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:13.223770 2995 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.017s sys 0.002s
I20260806 03:43:13.258776 3046 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:13.261837 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.261896 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.262683 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:13.269019 2554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.017s sys 0.041s
W20260806 03:43:13.299638 2554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.017s sys 0.041s
I20260806 03:43:13.280992 1664 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:13.265645 3031 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.225941 3001 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.300822 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.092800 2911 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.243824 3035 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:13.389322 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.286746 2648 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.259312 3055 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.329603 548 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
W20260806 03:43:13.290529 766 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.011s sys 0.058s
I20260806 03:43:13.329653 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:13.334447 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.344050 3025 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:13.343067 2927 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.423640 2927 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.423952 2648 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:13.348090 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:13.355782 3026 raft_consensus.cc:465] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Not starting pre-election -- already a leader
I20260806 03:43:13.303117 3046 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.370064 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:13.216802 3054 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.425495 674 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:13.427134 3054 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.322197 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.323064 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:13.267820 3029 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.243691 3015 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.416939 667 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:13.375427 3017 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:13.416548 766 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.011s sys 0.058s
I20260806 03:43:13.421484 3031 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.421044 3025 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.441977 1658 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:13.261155 3018 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.447419 659 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
W20260806 03:43:13.448067 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.018s user 0.021s sys 0.058s
I20260806 03:43:13.401890 3035 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:13.452543 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.022s user 0.021s sys 0.058s
I20260806 03:43:13.443091 3029 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.454324 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:13.448278 2761 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:13.458272 2761 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.458947 771 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:43:13.460305 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.461742 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.463177 771 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:13.428098 3046 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.464628 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:13.474524 562 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.009s sys 0.023s
W20260806 03:43:13.474979 562 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.009s sys 0.023s
I20260806 03:43:13.461789 2761 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:13.465545 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.04581e+06us
W20260806 03:43:13.482893 1665 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.016s sys 0.054s
I20260806 03:43:13.496557 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.504958 776 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:13.506696 3066 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260806 03:43:13.507337 3058 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3.
I20260806 03:43:13.507529 3025 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:13.509577 3068 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:13.510138 3068 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:13.510322 616 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:13.510754 3068 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.496369 3001 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.514173 3001 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.515375 3046 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:13.516014 3046 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:13.516423 3046 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:13.523553 767 raft_consensus.cc:2354] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4.
W20260806 03:43:13.523734 665 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.044s sys 0.036s
W20260806 03:43:13.524190 665 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.057s user 0.044s sys 0.036s
I20260806 03:43:13.526690 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:13.527557 674 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:13.536767 775 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.004s sys 0.017s
W20260806 03:43:13.537266 775 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.004s sys 0.017s
I20260806 03:43:13.545688 772 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
W20260806 03:43:13.498479 1665 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s user 0.016s sys 0.055s
I20260806 03:43:13.591898 501 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:13.597317 3018 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.603178 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.603955 668 raft_consensus.cc:2379] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:13.606057 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.594966 3030 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.606854 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.608153 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.608541 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.610543 668 raft_consensus.cc:2379] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:13.610543 561 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:13.612989 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.615504 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:13.615684 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.616632 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.617223 547 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:13.617146 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.619637 3056 consensus_queue.cc:1046] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.135s
I20260806 03:43:13.620684 3071 raft_consensus.cc:3032] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:13.620834 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.621057 3071 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:13.622006 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.624359 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.624435 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:13.599849 2997 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:13.632531 547 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 03:43:13.601426 3031 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:13.640244 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:13.635607 501 leader_election.cc:398] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4.
I20260806 03:43:13.641525 501 leader_election.cc:401] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:43:13.642860 771 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:13.644645 674 raft_consensus.cc:2379] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:13.614987 2828 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.005s sys 0.043s
W20260806 03:43:13.638434 2923 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.029s sys 0.065s
W20260806 03:43:13.649600 2700 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.007s
W20260806 03:43:13.655246 2828 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s user 0.005s sys 0.043s
I20260806 03:43:13.650185 3026 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.657711 3065 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.658411 2997 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:13.658068 3026 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.621891 3071 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:13.662906 669 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:13.637471 3015 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.659425 2997 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.666988 668 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:13.668885 2936 consensus_queue.cc:1046] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 03:43:13.670081 3026 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.646632 3031 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.671021 3071 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:13.647284 3035 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.626895 3030 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:13.677906 2923 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s user 0.029s sys 0.065s
I20260806 03:43:13.678718 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.688383 564 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
W20260806 03:43:13.688702 3019 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.375s user 0.008s sys 0.032s
W20260806 03:43:13.694198 3019 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.383s user 0.008s sys 0.033s
W20260806 03:43:13.682586 3041 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.008s sys 0.022s
I20260806 03:43:13.696645 3018 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.696964 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.697365 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:13.698598 403 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.000s sys 0.030s
W20260806 03:43:13.700495 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 485662us
W20260806 03:43:13.712661 769 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.037s sys 0.035s
I20260806 03:43:13.705734 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:13.719560 769 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s user 0.037s sys 0.036s
I20260806 03:43:13.720816 661 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:13.721720 664 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:13.711277 403 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.772s user 0.000s sys 0.030s
I20260806 03:43:13.692376 3026 raft_consensus.cc:3032] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:13.690294 3031 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:13.703438 3041 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s user 0.008s sys 0.022s
I20260806 03:43:13.715451 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:13.703548 773 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.000s sys 0.027s
I20260806 03:43:13.728217 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.730805 668 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:13.731132 562 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 2: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:13.732635 773 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s user 0.000s sys 0.027s
I20260806 03:43:13.755955 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 418) took 1036 ms. Trace:
I20260806 03:43:13.756403 663 rpcz_store.cc:276] 0806 03:43:12.719717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.766989 (+ 47272us) service_pool.cc:227] Handling call
0806 03:43:13.755330 (+988341us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321b264c6e 00007f321495b807 00007f321495b792 00007f321495b6c1 00007f321495b63c 00007f321495a8f2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:13.755865 (+ 535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2048260864}
I20260806 03:43:13.763913 3065 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4.
W20260806 03:43:13.765365 3082 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.066s user 0.001s sys 0.002s
I20260806 03:43:13.726840 3026 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:13.718796 1663 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:13.767225 3026 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.768455 3026 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.773741 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:13.775383 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 69727.8us
I20260806 03:43:13.786784 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.787588 499 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:13.791170 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:13.818022 547 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:13.817991 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:13.831061 1664 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:13.710939 3018 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.751531 2648 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.833462 615 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:13.783979 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:13.780848 3017 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:13.836141 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:13.833866 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:13.840518 1663 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:13.781998 2809 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.841635 2746 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:13.817039 2986 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 03:43:13.839222 2828 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:13.789299 3068 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.906883 2828 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.908335 3068 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.846323 1657 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 03:43:13.909018 2828 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.909685 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.851572 3058 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:13.910707 3058 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:13.911082 3058 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:13.911254 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:13.911935 1657 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:13.889333 2648 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:13.914268 2648 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.854089 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.854599 2936 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.915380 2746 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:13.915974 2936 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:13.901644 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 65593.6us
I20260806 03:43:13.909575 3046 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:13.850893 3018 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.847359 3030 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.919668 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.920986 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.921725 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.922113 3046 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.922664 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:13.929009 3060 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20260806 03:43:13.853322 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.851688 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:13.932396 775 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:13.933050 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:13.935045 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 167) took 1601 ms. Trace:
I20260806 03:43:13.935401 665 rpcz_store.cc:276] 0806 03:43:12.334142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.334745 (+ 603us) service_pool.cc:227] Handling call
0806 03:43:12.343557 (+ 8812us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:12.420540 (+ 76983us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:12.420573 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:12.428617 (+ 8044us) log.cc:842] Serialized 26 byte log entry
0806 03:43:13.524534 (+1095917us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:13.524569 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:13.524668 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:13.928401 (+403733us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:13.930099 (+ 1698us) raft_consensus.cc:1671] finished
0806 03:43:13.930118 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:13.934958 (+ 4840us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46170,"raft.queue_time_us":100908,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":11691392,"thread_start_us":1133617,"threads_started":2,"wal-append.queue_time_us":1262524}
I20260806 03:43:13.937067 499 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:13.939055 499 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:13.939381 501 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:13.915759 1670 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:13.941181 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 457226us
I20260806 03:43:13.852847 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:13.852110 3035 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.851868 2833 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.973047 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:13.971078 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 260522us
I20260806 03:43:13.940436 3030 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:13.985850 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:13.930475 2969 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:13.992789 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:13.930694 3048 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:13.995783 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 161971us
I20260806 03:43:13.996757 614 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:13.999621 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
W20260806 03:43:13.998315 730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.009s sys 0.056s
W20260806 03:43:14.002635 730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.009s sys 0.057s
I20260806 03:43:13.933018 2936 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:13.934512 3046 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:14.013303 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:14.062920 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 70226.8us
W20260806 03:43:14.063213 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 66775.3us
W20260806 03:43:14.063125 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 47267.9us
I20260806 03:43:13.941167 3066 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:14.071518 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 132237us
W20260806 03:43:14.072513 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 59293.2us
I20260806 03:43:13.937986 2809 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.003167 2707 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:14.104079 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:14.105705 614 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:14.106794 3048 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:14.076078 3030 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:14.077692 3046 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.067823 3001 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:13.992520 1658 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:14.107256 3098 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:14.016330 2969 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.106559 3066 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.109416 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:14.070887 3004 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:14.117578 1660 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:14.118884 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1879) took 1163 ms. Trace:
I20260806 03:43:14.062415 2987 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:14.121232 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.016108 3056 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:13.999583 3017 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:14.003288 2911 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:13.982393 3035 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.208341 3035 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:14.209154 3066 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:13.998315 2833 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.210748 2969 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:14.211308 3046 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.179298 564 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 03:43:14.183940 771 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:43:14.184015 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:14.140806 2828 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:14.129827 2746 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:14.194670 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 90969.5us
I20260806 03:43:14.190106 769 rpcz_store.cc:276] 0806 03:43:12.955714 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.958671 (+ 2957us) service_pool.cc:227] Handling call
0806 03:43:12.959066 (+ 395us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:12.971461 (+ 12395us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:12.971515 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:12.972014 (+ 499us) log.cc:842] Serialized 26 byte log entry
0806 03:43:13.719946 (+747932us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:13.719984 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:13.720060 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.118602 (+398542us) raft_consensus.cc:1671] finished
0806 03:43:14.118630 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:14.118797 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11891,"raft.queue_time_us":12710,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"thread_start_us":736402,"threads_started":1,"wal-append.queue_time_us":1083015}
W20260806 03:43:14.183348 766 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.022s sys 0.027s
I20260806 03:43:14.136199 3030 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:14.144100 2648 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:14.201471 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.197439 3065 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:14.224745 558 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:14.202955 1657 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:14.139936 3001 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:14.213150 3019 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:14.122551 2707 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:14.214892 2849 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:14.243112 728 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:14.213028 2949 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20260806 03:43:14.217686 3100 log.cc:924] Time spent T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260806 03:43:14.213093 3110 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260806 03:43:14.228020 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:14.252746 498 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:14.199241 3056 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:14.143312 3071 consensus_queue.cc:1046] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260806 03:43:14.214913 562 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:14.224355 766 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.023s sys 0.027s
I20260806 03:43:14.232451 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.213063 2833 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:14.234061 3001 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:14.227566 3030 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:14.274700 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.275904 558 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 2: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:14.277522 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:14.261190 2554 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:14.234328 2746 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:14.247324 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.264775 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:14.266094 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.254551 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.262912 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.227131 2828 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.288236 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:14.288849 775 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:43:14.263019 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.269824 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.233642 3005 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:14.242348 2707 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:14.224454 2987 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:14.196411 3098 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.285774 2554 raft_consensus.cc:686] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:14.262802 3056 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:14.263986 2809 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.264433 3026 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:14.251665 3107 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:14.306484 564 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:14.274622 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.253422 2849 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.309239 2554 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.313913 562 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:14.307598 771 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:14.330051 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.331122 499 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:14.334599 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.325861 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:14.324045 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.279874 2833 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:14.297425 2746 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.309273 3018 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:14.288108 3105 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:14.243323 3019 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.278245 3035 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.052s
I20260806 03:43:14.310715 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.332346 3046 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:14.391659 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 60640.2us
I20260806 03:43:14.396799 775 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
W20260806 03:43:14.382951 3082 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260806 03:43:14.387595 3134 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:14.382754 3107 raft_consensus.cc:686] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:14.408419 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 101935us
W20260806 03:43:14.411441 3114 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260806 03:43:14.395905 2911 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:14.309676 3017 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.393465 3046 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:14.371476 2833 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.420318 614 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:14.386967 3018 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
W20260806 03:43:14.412024 3108 log.cc:924] Time spent T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260806 03:43:14.419546 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:14.426407 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:14.426298 661 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:14.406348 3026 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:14.431442 2836 raft_consensus.cc:3032] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:14.411990 1663 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:14.435860 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:14.443516 3094 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:14.444185 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:14.444380 3117 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 03:43:14.416312 2911 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:14.420644 3046 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:14.416905 2987 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.413870 3029 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:14.482774 2828 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:14.487044 3099 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.247s user 0.000s sys 0.001s
I20260806 03:43:14.487910 3107 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.491396 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:14.479609 2833 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.498725 664 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:14.495127 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.530483 615 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:14.536266 772 raft_consensus.cc:1261] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:14.539714 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:14.538517 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 153304us
I20260806 03:43:14.552770 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:14.488817 2648 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:14.553818 2648 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:14.544986 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53662.6us
I20260806 03:43:14.498725 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.539228 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.548996 669 raft_consensus.cc:1261] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:14.538699 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:14.498056 2836 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:14.561911 2554 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:14.536968 2911 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:14.564822 547 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:14.565444 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:14.562479 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 256699us
I20260806 03:43:14.566884 2554 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:14.570896 616 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:14.534092 2809 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:14.574712 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:14.575989 616 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:14.549324 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.598994 2554 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.599769 2554 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:14.557695 2986 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:14.603539 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.569427 2833 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:14.572903 3143 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:14.574096 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1073) took 1040 ms. Trace:
I20260806 03:43:14.580276 2828 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:14.624845 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.606801 3019 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.635632 669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2018f00b21544838b05e402c218b85ea"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:14.612659 2849 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:14.641782 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:14.645747 1670 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:14.650961 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:14.639431 3058 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.728s user 0.006s sys 0.009s
I20260806 03:43:14.624239 2746 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.646108 542 raft_consensus.cc:2354] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:14.628675 3098 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:14.662415 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 1982) took 2263 ms. Trace:
W20260806 03:43:14.661106 3153 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.122s user 0.002s sys 0.000s
I20260806 03:43:14.659150 2836 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:14.669863 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 249659us
W20260806 03:43:14.673151 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 103581us
W20260806 03:43:14.674489 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 105949us
W20260806 03:43:14.678486 3125 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
I20260806 03:43:14.614493 3026 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.680838 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:14.681866 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:14.663877 671 rpcz_store.cc:276] 0806 03:43:12.394414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.416827 (+ 22413us) service_pool.cc:227] Handling call
0806 03:43:12.430737 (+ 13910us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:13.215867 (+785130us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.218853 (+ 2986us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:13.349093 (+130240us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b2c00000238. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:43:13.349737 (+ 644us) log.cc:842] Serialized 26 byte log entry
0806 03:43:13.701730 (+351993us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:14.129649 (+427919us) log.cc:842] Serialized 14 byte log entry
0806 03:43:14.137896 (+ 8247us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.137977 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.650784 (+512807us) raft_consensus.cc:1671] finished
0806 03:43:14.657389 (+ 6605us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:14.657638 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39106,"prepare.queue_time_us":509951,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":153112,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":13113,"spinlock_wait_cycles":13801728,"thread_start_us":1538474,"threads_started":3,"wal-append.queue_time_us":718601}
I20260806 03:43:14.687216 3156 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:14.704921 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:14.710201 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:14.688720 3018 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:14.662675 3058 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:15.218238 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.059s sys 0.155s
I20260806 03:43:14.703924 3004 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:14.621783 766 rpcz_store.cc:276] 0806 03:43:13.529986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:13.531904 (+ 1918us) service_pool.cc:227] Handling call
0806 03:43:13.641304 (+109400us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:13.679335 (+ 38031us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.679370 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:13.680682 (+ 1312us) log.cc:842] Serialized 590 byte log entry
0806 03:43:14.273884 (+593202us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:14.273930 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.278244 (+ 4314us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.549393 (+271149us) raft_consensus.cc:1671] finished
0806 03:43:14.549417 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:14.569895 (+ 20478us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1647,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":582699,"threads_started":1,"wal-append.queue_time_us":590511}
W20260806 03:43:15.141691 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.078s sys 0.128s
W20260806 03:43:15.221628 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s user 0.060s sys 0.156s
W20260806 03:43:15.227773 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s user 0.081s sys 0.160s
I20260806 03:43:14.770838 546 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:15.228606 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:15.213819 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:14.917225 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 141402us
I20260806 03:43:15.219902 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:15.232548 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:15.230078 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:14.719924 3121 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.244s user 0.001s sys 0.001s
I20260806 03:43:15.251298 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:15.252279 546 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:15.269374 3149 raft_consensus.cc:2741] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
W20260806 03:43:14.932464 3148 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.442s user 0.003s sys 0.002s
W20260806 03:43:14.852468 3085 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.720s user 0.001s sys 0.003s
W20260806 03:43:14.933354 3048 consensus_meta.cc:287] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.797s user 0.010s sys 0.004s
W20260806 03:43:14.834709 3001 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.550s user 0.011s sys 0.002s
W20260806 03:43:14.846508 3147 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.275s user 0.002s sys 0.000s
W20260806 03:43:14.853606 3142 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
W20260806 03:43:14.812646 3122 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.319s user 0.001s sys 0.001s
W20260806 03:43:15.125777 2648 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.571s user 0.008s sys 0.001s
W20260806 03:43:15.124087 3056 consensus_meta.cc:287] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.780s user 0.004s sys 0.010s
W20260806 03:43:15.125734 3030 consensus_meta.cc:287] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.838s user 0.004s sys 0.008s
I20260806 03:43:15.283032 667 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:15.074245 3046 consensus_meta.cc:287] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.523s user 0.006s sys 0.007s
I20260806 03:43:14.719558 3124 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:15.298745 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 60310.5us
I20260806 03:43:14.719429 3035 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:14.973698 3092 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:15.307871 2995 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:15.350296 2221 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:15.351987 2221 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.325012 3048 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:14.951567 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:15.253152 3134 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.837s user 0.007s sys 0.007s
I20260806 03:43:14.812156 3146 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:15.318292 659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.715s user 0.000s sys 0.025s
I20260806 03:43:14.704159 2707 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:14.934829 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 235) took 1238 ms. Trace:
I20260806 03:43:15.226660 2700 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.226445 3071 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:15.300011 2986 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.003s sys 0.027s
I20260806 03:43:15.232527 2761 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:15.309688 1660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.009s sys 0.025s
I20260806 03:43:15.232767 3105 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.182925 2949 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.298209 3004 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:15.353041 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:15.341003 3156 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:15.358858 2828 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:15.359088 3058 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.347692 3151 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:15.376346 3134 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:15.298800 2907 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 03:43:15.373627 614 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:15.451536 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 78015.7us
W20260806 03:43:15.374635 659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s user 0.000s sys 0.026s
I20260806 03:43:15.454936 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 1985) took 1932 ms. Trace:
I20260806 03:43:15.455467 659 rpcz_store.cc:276] 0806 03:43:13.522085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:13.524328 (+ 2243us) service_pool.cc:227] Handling call
0806 03:43:13.536697 (+ 12369us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:13.540610 (+ 3913us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.540656 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:13.541180 (+ 524us) log.cc:842] Serialized 26 byte log entry
0806 03:43:14.003780 (+462600us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:14.602667 (+598887us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b18002db020. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f3215fae82c 00007f32133d99dc 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f3219cf4167 00007f3216329807 00007f3216338f0f
0806 03:43:14.602815 (+ 148us) log.cc:842] Serialized 14 byte log entry
0806 03:43:14.956353 (+353538us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b48001a01d8. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:43:15.454492 (+498139us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.454606 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:15.454671 (+ 65us) raft_consensus.cc:1671] finished
0806 03:43:15.454689 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.454859 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":406596,"prepare.queue_time_us":208367,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":33054,"raft.run_cpu_time_us":2097,"raft.run_wall_time_us":369229,"spinlock_wait_cycles":14069376,"thread_start_us":1405754,"threads_started":3,"wal-append.queue_time_us":633302}
I20260806 03:43:15.375620 2221 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.458915 2221 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:15.459385 2221 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:15.459757 2221 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:15.335314 3066 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:15.398425 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 45454.4us
I20260806 03:43:15.220360 2836 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:15.330085 3001 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.401187 775 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:15.377512 2986 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s user 0.003s sys 0.028s
I20260806 03:43:15.364615 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1632) took 1748 ms. Trace:
W20260806 03:43:15.360852 2911 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.779s user 0.006s sys 0.006s
I20260806 03:43:15.442404 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2001) took 1024 ms. Trace:
I20260806 03:43:15.354202 3030 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:15.393862 1660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.009s sys 0.025s
I20260806 03:43:15.367795 3092 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:15.362234 3018 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:15.338929 3056 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.340466 2648 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.353899 3046 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.492925 561 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.132s user 0.004s sys 0.035s
W20260806 03:43:15.497116 561 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.154s user 0.004s sys 0.036s
I20260806 03:43:15.368031 3035 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.472572 667 rpcz_store.cc:276] 0806 03:43:14.418414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.419259 (+ 845us) service_pool.cc:227] Handling call
0806 03:43:15.442132 (+1022873us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54002ae754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:15.442300 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1834414464}
I20260806 03:43:15.486956 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:15.404466 3005 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:43:15.489135 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.908s user 0.004s sys 0.038s
I20260806 03:43:15.493721 1622 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:15.501150 2987 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.025s sys 0.012s
I20260806 03:43:15.404281 3146 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.394696 2700 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.503233 564 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:15.397430 2761 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.500990 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.920s user 0.004s sys 0.038s
I20260806 03:43:15.365779 2995 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.518123 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 937627us
W20260806 03:43:15.504027 2987 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.025s sys 0.013s
I20260806 03:43:15.512058 2221 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
W20260806 03:43:15.514863 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.019s sys 0.022s
W20260806 03:43:15.520566 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.872s user 0.019s sys 0.022s
W20260806 03:43:15.521020 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 876173us
I20260806 03:43:15.522113 615 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:15.501699 3165 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:15.535797 615 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:15.531808 2221 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
W20260806 03:43:15.531985 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 290756us
I20260806 03:43:15.480060 2986 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:15.400398 3071 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.538314 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 251) took 2531 ms. Trace:
I20260806 03:43:15.496553 3159 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.542014 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:15.544804 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:15.538869 773 rpcz_store.cc:276] 0806 03:43:12.987478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.987643 (+ 165us) service_pool.cc:227] Handling call
0806 03:43:13.001253 (+ 13610us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:13.001683 (+ 430us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.001715 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:13.002475 (+ 760us) log.cc:842] Serialized 26 byte log entry
0806 03:43:13.733123 (+730648us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:13.733179 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:13.733272 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.848760 (+1115488us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:15.504957 (+656197us) raft_consensus.cc:1671] finished
0806 03:43:15.504980 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.518543 (+ 13563us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1518,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":3665,"spinlock_wait_cycles":5930240,"thread_start_us":730084,"threads_started":1,"wal-append.queue_time_us":995080}
I20260806 03:43:15.502089 3019 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:15.404083 2949 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.495097 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 241) took 1571 ms. Trace:
I20260806 03:43:15.550932 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:15.553779 659 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:15.507174 3054 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:15.503091 3005 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:15.556528 547 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:43:15.475260 3066 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.473825 2828 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.399888 3105 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.510679 2969 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.036s
I20260806 03:43:15.487876 2997 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260806 03:43:15.491227 3018 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.489266 2911 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.582342 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:15.535286 3017 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s user 0.018s sys 0.026s
I20260806 03:43:15.605559 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:15.558297 3005 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:15.625159 563 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:15.432083 3156 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.573388 2809 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.012s sys 0.033s
W20260806 03:43:15.592489 2969 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.001s sys 0.036s
W20260806 03:43:15.604542 3107 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.012s sys 0.053s
I20260806 03:43:15.609277 1658 raft_consensus.cc:2354] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
W20260806 03:43:15.644322 3107 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.952s user 0.012s sys 0.054s
I20260806 03:43:15.637580 547 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:15.615273 3017 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.018s sys 0.026s
I20260806 03:43:15.554697 3159 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:15.651945 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.033s sys 0.065s
I20260806 03:43:15.650180 614 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:15.658429 667 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 03:43:15.660334 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.033s sys 0.065s
I20260806 03:43:15.664911 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1916) took 1136 ms. Trace:
I20260806 03:43:15.665385 775 rpcz_store.cc:276] 0806 03:43:14.528235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.528457 (+ 222us) service_pool.cc:227] Handling call
0806 03:43:14.528810 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:15.471946 (+943136us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.664693 (+192747us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b5400311354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:15.664858 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2005867264}
I20260806 03:43:15.672338 3160 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:15.602344 3168 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260806 03:43:15.635067 3065 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:15.685573 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:15.702450 775 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:15.567912 3019 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:15.642696 2809 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.035s user 0.012s sys 0.033s
I20260806 03:43:15.554908 3092 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.696560 3146 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.653465 2707 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.699204 771 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:15.479657 3151 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.649049 2949 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.673471 3160 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
W20260806 03:43:15.703398 1622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.968s (SENT) [suppressed 30 similar messages]
I20260806 03:43:15.701136 3048 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:15.371605 1662 rpcz_store.cc:276] 0806 03:43:13.696089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:13.696917 (+ 828us) service_pool.cc:227] Handling call
0806 03:43:13.697317 (+ 400us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:13.697714 (+ 397us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.697744 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:13.698383 (+ 639us) log.cc:842] Serialized 52 byte log entry
0806 03:43:13.773049 (+ 74666us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f32133d95fe 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb
0806 03:43:14.193838 (+420789us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:14.203885 (+ 10047us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.217929 (+ 14044us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.916238 (+698309us) raft_consensus.cc:1671] finished
0806 03:43:14.917966 (+ 1728us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:14.934721 (+ 16755us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3500,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5871,"spinlock_wait_cycles":9726848,"thread_start_us":444961,"threads_started":1,"wal-append.queue_time_us":604615}
I20260806 03:43:15.573361 3165 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:15.712157 2809 raft_consensus.cc:686] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:15.479840 548 rpcz_store.cc:276] 0806 03:43:13.600630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:13.601276 (+ 646us) service_pool.cc:227] Handling call
0806 03:43:13.663697 (+ 62421us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:13.706675 (+ 42978us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:13.711253 (+ 4578us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:13.758338 (+ 47085us) log.cc:842] Serialized 26 byte log entry
0806 03:43:14.233635 (+475297us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:14.325074 (+ 91439us) log.cc:842] Serialized 14 byte log entry
0806 03:43:14.408809 (+ 83735us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.410602 (+ 1793us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:14.836373 (+425771us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b4800150258. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:43:15.237730 (+401357us) raft_consensus.cc:1671] finished
0806 03:43:15.332033 (+ 94303us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.349241 (+ 17208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175163,"prepare.queue_time_us":111732,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":86917,"raft.run_cpu_time_us":3392,"raft.run_wall_time_us":436742,"spinlock_wait_cycles":7762688,"thread_start_us":437523,"threads_started":1,"wal-append.queue_time_us":563759}
I20260806 03:43:15.718185 3029 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:15.825704 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.040s sys 0.035s
W20260806 03:43:15.847812 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.042s sys 0.036s
I20260806 03:43:15.822882 3095 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:15.848325 3095 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:15.848691 3095 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:15.848881 3105 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.656498 2986 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:15.821810 3172 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260806 03:43:15.807708 3183 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.108s user 0.001s sys 0.001s
I20260806 03:43:15.880649 667 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:15.801131 3030 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.868814 3071 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.870101 615 leader_election.cc:398] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:15.870926 772 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260806 03:43:15.804153 2648 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.803975 3056 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.857692 3035 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.841288 3048 raft_consensus.cc:686] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:15.890904 772 pending_rounds.cc:76] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 4
W20260806 03:43:15.884970 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 344997us
I20260806 03:43:15.893993 615 leader_election.cc:401] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:43:15.842842 3046 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:15.851486 3107 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.829401 2828 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:15.869941 2746 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:15.881551 3177 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.278s user 0.002s sys 0.000s
I20260806 03:43:15.905578 772 pending_rounds.cc:97] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 03:43:15.924594 772 raft_consensus.cc:2866] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:15.872164 3001 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:15.923450 3018 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:15.938473 2746 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:15.914780 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 50437.1us
W20260806 03:43:15.918648 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.012s sys 0.007s
W20260806 03:43:15.942263 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.711s user 0.012s sys 0.007s
W20260806 03:43:15.942734 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 711660us
W20260806 03:43:15.920341 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 216965us
I20260806 03:43:15.897699 2911 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:15.884372 2997 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:15.882654 3179 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:15.870491 3029 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.905555 2995 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:15.898051 3019 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:15.957651 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 271406us
I20260806 03:43:15.922672 3035 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:15.937405 3048 consensus_queue.cc:237] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:15.881182 3174 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:15.555980 1665 rpcz_store.cc:276] 0806 03:43:13.917551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:13.925006 (+ 7455us) service_pool.cc:227] Handling call
0806 03:43:13.925414 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:14.061454 (+136040us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:14.064995 (+ 3541us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:14.200037 (+135042us) log.cc:842] Serialized 26 byte log entry
0806 03:43:14.377781 (+177744us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:14.382508 (+ 4727us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.384701 (+ 2193us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:15.461820 (+1077119us) raft_consensus.cc:1671] finished
0806 03:43:15.461850 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.489140 (+ 27290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50883,"raft.queue_time_us":70504,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":63900,"thread_start_us":62961,"threads_started":1,"wal-append.queue_time_us":203967}
I20260806 03:43:15.973459 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:15.965464 3182 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.176s user 0.000s sys 0.002s
I20260806 03:43:15.922825 3066 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:15.981767 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2009) took 1438 ms. Trace:
W20260806 03:43:15.943565 3171 log.cc:924] Time spent T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20260806 03:43:15.988755 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:15.998096 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:15.988808 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:15.989878 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 979) took 1742 ms. Trace:
I20260806 03:43:15.990512 547 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:15.999728 668 rpcz_store.cc:276] 0806 03:43:14.543668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.543886 (+ 218us) service_pool.cc:227] Handling call
0806 03:43:14.544299 (+ 413us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:14.544552 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:14.544593 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:14.555702 (+ 11109us) log.cc:842] Serialized 26 byte log entry
0806 03:43:14.685352 (+129650us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:14.685387 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:14.685448 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:15.981357 (+1295909us) raft_consensus.cc:1671] finished
0806 03:43:15.981380 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.981703 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":86635,"threads_started":1,"wal-append.queue_time_us":814731}
I20260806 03:43:16.002250 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:15.959532 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:16.006759 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 47371.3us
I20260806 03:43:16.007586 727 leader_election.cc:398] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:16.007930 727 leader_election.cc:401] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:15.946750 3001 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.942330 3190 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.948045 3056 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:15.992798 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:15.947436 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:15.919085 3107 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.003222 772 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:15.980803 2828 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:15.984858 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 466) took 1542 ms. Trace:
I20260806 03:43:15.958989 3094 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260806 03:43:15.895612 3186 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.116s
I20260806 03:43:15.988322 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:15.996029 2907 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:16.044049 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:15.993273 3030 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:15.925426 2809 consensus_queue.cc:237] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.046018 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.014588 2949 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:15.972560 2997 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:15.981302 3179 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.048935 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:16.004338 561 rpcz_store.cc:276] 0806 03:43:14.247829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.247957 (+ 128us) service_pool.cc:227] Handling call
0806 03:43:14.255910 (+ 7953us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:14.256158 (+ 248us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:15.497459 (+1241301us) log.cc:842] Serialized 14 byte log entry
0806 03:43:15.982509 (+485050us) log.cc:842] Serialized 14 byte log entry
0806 03:43:15.987313 (+ 4804us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:15.989594 (+ 2281us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:15.989636 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.989697 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.989810 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76392,"prepare.queue_time_us":143641,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":1972,"raft.queue_time_us":59748,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":16149,"spinlock_wait_cycles":4480,"thread_start_us":1639102,"threads_started":3,"wal-append.queue_time_us":427394}
I20260806 03:43:16.192369 1670 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:16.060985 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:16.126093 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:16.014823 2707 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:16.183634 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.131283 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.026054 3159 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.018821 3185 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:16.025776 3098 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:16.159265 3199 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.211928 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:16.225008 771 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:16.035073 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.213660 3200 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.015475 3046 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:15.996753 3198 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.182214 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:16.211817 3041 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.045522 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:16.068150 3195 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:16.228658 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 98647.2us
I20260806 03:43:16.229820 547 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:16.049460 2828 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.234753 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:16.068374 3017 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:16.235898 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.234432 3066 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.224404 3160 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.240696 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.227751 3055 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.379858 3055 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.254288 771 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:16.046518 2969 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.259212 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 204) took 1744 ms. Trace:
I20260806 03:43:16.383244 564 rpcz_store.cc:276] 0806 03:43:14.494627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.494776 (+ 149us) service_pool.cc:227] Handling call
0806 03:43:16.238458 (+1743682us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54000221d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:16.238634 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187455360}
I20260806 03:43:16.273562 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 208) took 1734 ms. Trace:
I20260806 03:43:16.383359 2969 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.215754 3058 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.231555 3156 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.208015 2907 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.039670 3056 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.046773 3001 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.219601 3165 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.126736 3105 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:16.236392 3159 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:16.187295 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:16.251688 2707 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:16.034484 3190 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.217098 3030 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.231789 3098 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:16.356180 563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 203) took 1871 ms (client timeout 1968 ms). Trace:
I20260806 03:43:16.398677 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:16.753908 3194 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.502s user 0.001s sys 0.001s
I20260806 03:43:16.738822 3179 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.797389 3107 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.355063 2221 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.799734 2221 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:16.794092 563 rpcz_store.cc:269] 0806 03:43:14.484461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.484612 (+ 151us) service_pool.cc:227] Handling call
0806 03:43:16.355920 (+1871308us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001c7fd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:16.356106 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2466907904}
I20260806 03:43:16.719631 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:16.815423 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.385056 558 rpcz_store.cc:276] 0806 03:43:14.539067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.566963 (+ 27896us) service_pool.cc:227] Handling call
0806 03:43:14.598100 (+ 31137us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:14.598543 (+ 443us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:14.598583 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:14.611342 (+ 12759us) log.cc:842] Serialized 26 byte log entry
0806 03:43:15.146078 (+534736us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:15.632240 (+486162us) log.cc:842] Serialized 14 byte log entry
0806 03:43:15.632473 (+ 233us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.632553 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:16.273326 (+640773us) raft_consensus.cc:1671] finished
0806 03:43:16.273361 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:16.273492 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":709692,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":25624,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":9869,"thread_start_us":942974,"threads_started":2,"wal-append.queue_time_us":781477}
W20260806 03:43:16.826151 665 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.004s sys 0.015s
W20260806 03:43:16.827366 665 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.005s sys 0.015s
I20260806 03:43:16.829183 2221 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.829869 2221 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:16.781014 2836 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.299s user 0.014s sys 0.001s
I20260806 03:43:16.831259 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:16.836127 772 raft_consensus.cc:2379] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:16.779948 3217 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20260806 03:43:16.253441 3019 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:16.736734 3215 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:16.840256 1670 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 03:43:16.675225 3171 log.cc:924] Time spent T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
W20260806 03:43:16.737864 3193 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.745s user 0.001s sys 0.002s
I20260806 03:43:16.862236 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1922) took 1562 ms. Trace:
I20260806 03:43:16.259943 2949 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.275694 3094 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.274888 2911 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.336709 2746 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.271648 3199 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:16.816888 3196 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.510s user 0.061s sys 0.118s
I20260806 03:43:16.261364 3151 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.371835 3174 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.538048 1662 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:16.737290 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:16.485471 3029 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.370805 548 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:16.484737 2828 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.824522 3058 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.867460 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.870146 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:16.862931 768 rpcz_store.cc:276] 0806 03:43:15.300295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.300454 (+ 159us) service_pool.cc:227] Handling call
0806 03:43:15.300777 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:15.301086 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:15.301113 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:15.329558 (+ 28445us) log.cc:842] Serialized 26 byte log entry
0806 03:43:15.669032 (+339474us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:15.669068 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.669147 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:16.861847 (+1192700us) raft_consensus.cc:1671] finished
0806 03:43:16.861877 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:16.862136 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96485,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":60820,"thread_start_us":301410,"threads_started":1,"wal-append.queue_time_us":528760}
I20260806 03:43:16.834807 2707 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.480247 3207 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:16.273294 3046 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.814152 3159 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.842597 3165 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:16.277827 3198 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.883250 2221 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.841056 2997 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.368937 3041 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.491191 3146 consensus_queue.cc:1046] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.375s
I20260806 03:43:16.436977 3055 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.824163 3066 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.823092 3105 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:16.868837 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.052s sys 0.065s
W20260806 03:43:16.946707 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.052s sys 0.066s
I20260806 03:43:16.818864 3017 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.231416 3185 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.877838 2836 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:16.962476 2836 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.858726 3019 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:16.906404 3095 consensus_meta.cc:287] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.053s user 0.013s sys 0.004s
I20260806 03:43:16.332396 3200 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.937541 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:16.878870 2949 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.864262 3186 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 03:43:16.953241 659 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:16.942857 3190 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.959519 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2014) took 1478 ms. Trace:
I20260806 03:43:16.960990 3017 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.965133 2221 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.947059 2969 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.043777 1660 rpcz_store.cc:276] 0806 03:43:14.438820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.441050 (+ 2230us) service_pool.cc:227] Handling call
0806 03:43:14.441647 (+ 597us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:14.549612 (+107965us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:14.555905 (+ 6293us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:14.604894 (+ 48989us) log.cc:842] Serialized 26 byte log entry
0806 03:43:14.956780 (+351886us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b1800166480. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:15.494750 (+537970us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:15.502878 (+ 8128us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:15.511664 (+ 8786us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:15.941095 (+429431us) raft_consensus.cc:1671] finished
0806 03:43:15.943883 (+ 2788us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:15.981091 (+ 37208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2409,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":160,"spinlock_wait_cycles":9641856,"thread_start_us":938502,"threads_started":2,"wal-append.queue_time_us":912314}
I20260806 03:43:16.990937 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.961015 3058 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.949556 2761 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.997653 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.997880 3200 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:16.999248 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:16.999961 664 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:17.003760 3186 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.003975 3046 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:16.987605 3035 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.996s user 0.011s sys 0.005s
I20260806 03:43:16.982203 665 rpcz_store.cc:276] 0806 03:43:15.481374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.486326 (+ 4952us) service_pool.cc:227] Handling call
0806 03:43:15.505038 (+ 18712us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:15.702493 (+197455us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:15.971285 (+268792us) log.cc:842] Serialized 14 byte log entry
0806 03:43:16.957944 (+986659us) spinlock_profiling.cc:243] Waited 8.63 ms on lock 0x7b18000a04e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f3215fae82c 00007f32133d99dc 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7
0806 03:43:16.958079 (+ 135us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.958141 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:16.958243 (+ 102us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:16.959224 (+ 981us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540015b8d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:16.959281 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:16.959434 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11004,"prepare.queue_time_us":552555,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":600216,"raft.run_wall_time_us":9145,"spinlock_wait_cycles":504223104,"thread_start_us":1089573,"threads_started":3}
I20260806 03:43:16.937440 3207 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.011850 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:17.012514 665 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:17.012740 2949 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.943477 3165 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:16.975682 3198 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:16.395551 3195 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:16.965839 2997 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 03:43:16.992848 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.026134 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.010433 3105 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.027370 3019 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.029191 2707 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.030892 3030 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.034444 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:16.924762 3174 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.037245 1670 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:16.878298 3215 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.061707 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:17.061839 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1570) took 1046 ms. Trace:
I20260806 03:43:17.019852 3035 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:17.032016 3066 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:16.878173 2911 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.047812 659 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:17.004666 3186 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.019029 2907 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.059818 3199 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.041218 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 553) took 1042 ms. Trace:
I20260806 03:43:17.062363 665 rpcz_store.cc:276] 0806 03:43:16.014972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.013187 (+998215us) service_pool.cc:227] Handling call
0806 03:43:17.061755 (+ 48568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:17.069227 1668 rpcz_store.cc:276] 0806 03:43:15.999100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.999250 (+ 150us) service_pool.cc:227] Handling call
0806 03:43:16.022782 (+ 23532us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.275291 (+252509us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.275337 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.275417 (+ 80us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:16.275445 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.040865 (+765420us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54006414d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.040938 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.041129 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":580096384}
I20260806 03:43:17.070149 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:17.070845 665 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:17.071508 3207 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.072791 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.051827 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.054355 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.074744 664 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:17.066835 3030 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:17.084584 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1093) took 1079 ms. Trace:
I20260806 03:43:17.107365 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1571) took 1092 ms. Trace:
I20260806 03:43:16.978442 3095 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.086766 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.116930 664 rpcz_store.cc:276] 0806 03:43:16.015268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.998945 (+983677us) service_pool.cc:227] Handling call
0806 03:43:17.107176 (+108231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:17.048235 3056 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.127111 1664 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:17.111197 769 rpcz_store.cc:276] 0806 03:43:16.005409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.007029 (+ 1620us) service_pool.cc:227] Handling call
0806 03:43:16.044901 (+ 37872us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.090522 (+ 45621us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.090555 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.090606 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:16.090633 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.084190 (+993557us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b54002f5654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.084251 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.084513 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104332160}
I20260806 03:43:17.116709 3030 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.128584 3030 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:17.129187 3188 log.cc:924] Time spent T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 1.325s user 0.010s sys 0.007s
I20260806 03:43:17.083105 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1562) took 1538 ms. Trace:
I20260806 03:43:16.877727 3094 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.089453 3041 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.138674 2836 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.139688 2836 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:17.115917 3005 consensus_meta.cc:287] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.480s user 0.015s sys 0.004s
I20260806 03:43:17.140005 3185 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.147194 2836 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.150331 3159 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:17.129361 3124 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.775s user 0.020s sys 0.000s
I20260806 03:43:17.074739 3001 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.155539 3215 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.129766 664 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:43:17.163837 3041 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.134073 674 rpcz_store.cc:276] 0806 03:43:15.544606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.544708 (+ 102us) service_pool.cc:227] Handling call
0806 03:43:15.544962 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:15.802098 (+257136us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:15.802131 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:15.802183 (+ 52us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:15.802224 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.082911 (+1280687us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54000b1154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.082948 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.083045 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1975,"raft.queue_time_us":1706,"raft.run_cpu_time_us":360,"raft.run_wall_time_us":50216,"spinlock_wait_cycles":587652864}
I20260806 03:43:17.166780 3041 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.167901 3186 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.167906 2911 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.171566 3174 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.158294 3005 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.161916 3124 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:17.187654 775 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.046s sys 0.061s
W20260806 03:43:17.188086 775 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.968s user 0.046s sys 0.061s
W20260806 03:43:17.189033 615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.008s sys 0.051s
W20260806 03:43:17.189390 615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.960s user 0.008s sys 0.051s
I20260806 03:43:17.184597 667 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:17.203343 671 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:17.204486 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.029s sys 0.039s
W20260806 03:43:17.204831 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.950s user 0.029s sys 0.039s
W20260806 03:43:17.205093 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 988035us
I20260806 03:43:17.227427 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.209807 3215 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:17.245702 3095 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.245533 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.245869 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.247076 1665 raft_consensus.cc:2396] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:17.249262 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:17.253094 3176 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 1.312s user 0.004s sys 0.008s
I20260806 03:43:17.255568 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1931) took 1125 ms. Trace:
I20260806 03:43:17.256083 773 rpcz_store.cc:276] 0806 03:43:16.129832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.130140 (+ 308us) service_pool.cc:227] Handling call
0806 03:43:16.196289 (+ 66149us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.951843 (+755554us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.951885 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.951942 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:16.951985 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.255280 (+303295us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b5400071054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.255341 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.255495 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1736944384}
I20260806 03:43:17.250044 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.250234 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.214530 3165 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.265872 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:17.267504 3092 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:17.210561 3195 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.270812 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.271656 776 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
W20260806 03:43:17.235927 3018 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.977s user 0.014s sys 0.026s
W20260806 03:43:17.346621 3018 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.115s user 0.015s sys 0.026s
I20260806 03:43:17.237062 3185 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:17.347407 3185 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.317101 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 273) took 1793 ms. Trace:
I20260806 03:43:17.318145 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1569) took 1303 ms. Trace:
I20260806 03:43:17.354321 659 rpcz_store.cc:276] 0806 03:43:16.014712 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.212077 (+197365us) service_pool.cc:227] Handling call
0806 03:43:17.317907 (+1105830us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540019afd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:17.318036 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1428560768}
I20260806 03:43:17.356050 659 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:17.319411 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.323972 3107 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:17.324182 3005 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.324872 3094 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.287902 3149 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:17.329810 3206 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:17.382196 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.383963 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.269507 3215 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.386871 1664 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:17.388218 3215 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:17.294480 3041 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.391999 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 186) took 1394 ms. Trace:
I20260806 03:43:17.392508 667 rpcz_store.cc:276] 0806 03:43:15.997671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.997866 (+ 195us) service_pool.cc:227] Handling call
0806 03:43:17.391756 (+1393890us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002ae754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:17.391899 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2453087616}
I20260806 03:43:17.396025 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 214) took 1270 ms. Trace:
I20260806 03:43:17.344352 499 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:17.345019 665 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:17.396524 562 rpcz_store.cc:276] 0806 03:43:16.137331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.210712 (+ 73381us) service_pool.cc:227] Handling call
0806 03:43:16.211198 (+ 486us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.963923 (+752725us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.963980 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.964068 (+ 88us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:16.964120 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.395719 (+431599us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b5400022954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.395779 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.395948 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1730426752}
I20260806 03:43:17.336614 1665 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:17.373970 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.394286 667 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:17.322530 2700 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.375227 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2022) took 1153 ms. Trace:
W20260806 03:43:17.380688 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.051s user 0.011s sys 0.049s
I20260806 03:43:17.398795 659 rpcz_store.cc:276] 0806 03:43:16.221595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.355405 (+1133810us) service_pool.cc:227] Handling call
0806 03:43:17.355832 (+ 427us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.374908 (+ 19076us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.375152 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:17.399641 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.070s user 0.012s sys 0.050s
I20260806 03:43:17.296043 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:17.396510 1664 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:17.403187 674 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:17.404915 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1563) took 1859 ms. Trace:
I20260806 03:43:17.397583 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1573) took 1381 ms. Trace:
I20260806 03:43:17.405409 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2018) took 1220 ms. Trace:
I20260806 03:43:17.405964 674 rpcz_store.cc:276] 0806 03:43:16.185011 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.166321 (+981310us) service_pool.cc:227] Handling call
0806 03:43:17.402654 (+236333us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:17.402943 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.404945 (+ 2002us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.405341 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11846912}
I20260806 03:43:17.408022 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1567) took 1397 ms. Trace:
I20260806 03:43:17.408500 668 rpcz_store.cc:276] 0806 03:43:16.010777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.010964 (+ 187us) service_pool.cc:227] Handling call
0806 03:43:16.046790 (+ 35826us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.964482 (+917692us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.964523 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:16.964565 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:16.964600 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.407747 (+443147us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54002aec54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.407805 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.407956 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2110082944}
I20260806 03:43:17.405429 671 rpcz_store.cc:276] 0806 03:43:15.545440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.547676 (+ 2236us) service_pool.cc:227] Handling call
0806 03:43:15.568089 (+ 20413us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.209874 (+1641785us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.404598 (+194724us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540015b3d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.404839 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3760123776}
W20260806 03:43:17.418030 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 73794.5us
I20260806 03:43:17.405900 665 rpcz_store.cc:276] 0806 03:43:16.016445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.069768 (+1053323us) service_pool.cc:227] Handling call
0806 03:43:17.397519 (+327751us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:17.414772 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 473) took 1465 ms. Trace:
I20260806 03:43:17.423108 1671 rpcz_store.cc:276] 0806 03:43:15.915688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.915785 (+ 97us) service_pool.cc:227] Handling call
0806 03:43:17.319228 (+1403443us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:17.380768 (+ 61540us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13517056}
I20260806 03:43:17.427007 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.376046 3107 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:17.423979 3206 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:17.432355 2872 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:17.433409 2872 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:17.267518 3178 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:17.433764 2872 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:17.433822 3206 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.435524 3206 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:17.427337 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 131385us
I20260806 03:43:17.447268 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2024) took 1209 ms. Trace:
I20260806 03:43:17.448276 667 rpcz_store.cc:276] 0806 03:43:16.237378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.393451 (+1156073us) service_pool.cc:227] Handling call
0806 03:43:17.393867 (+ 416us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.398547 (+ 4680us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.447196 (+ 48649us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:17.425945 3005 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:17.429175 3035 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:17.436302 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 250) took 1887 ms. Trace:
I20260806 03:43:17.456542 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.429229 2836 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.453162 3005 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:17.464836 3005 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:17.465445 3098 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.414543 3149 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:17.485396 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.992s user 0.017s sys 0.031s
W20260806 03:43:17.486362 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.993s user 0.017s sys 0.032s
I20260806 03:43:17.493417 615 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:17.461611 3035 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.502810 1663 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:17.353935 770 rpcz_store.cc:276] 0806 03:43:15.524066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.530701 (+ 6635us) service_pool.cc:227] Handling call
0806 03:43:17.315705 (+1785004us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:17.316102 (+ 397us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.316388 (+ 286us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:17.316460 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:17.316572 (+ 112us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:17.316651 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.316787 (+ 136us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.317016 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10199424}
I20260806 03:43:17.504256 3035 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.505471 3149 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.505275 3030 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.506326 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.507524 3030 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:17.461488 1657 rpcz_store.cc:276] 0806 03:43:15.544844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.553403 (+ 8559us) service_pool.cc:227] Handling call
0806 03:43:15.635936 (+ 82533us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:15.766792 (+130856us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:15.783469 (+ 16677us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:15.800817 (+ 17348us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:15.804600 (+ 3783us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.423169 (+1618569us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x7b5400606b54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:17.425436 (+ 2267us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.432362 (+ 6926us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178515712}
I20260806 03:43:17.511379 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:17.513480 1664 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:17.514586 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.181s user 0.004s sys 0.037s
W20260806 03:43:17.514953 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.189s user 0.004s sys 0.037s
W20260806 03:43:17.515486 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.18986e+06us
W20260806 03:43:17.516444 546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.185s user 0.005s sys 0.045s
W20260806 03:43:17.516872 546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.185s user 0.005s sys 0.045s
I20260806 03:43:17.490878 3185 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:17.519064 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.011s sys 0.041s
I20260806 03:43:17.483644 3220 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:17.523662 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.383s user 0.011s sys 0.041s
W20260806 03:43:17.531214 2986 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.018s sys 0.024s
W20260806 03:43:17.536413 1625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.281s user 0.019s sys 0.073s
W20260806 03:43:17.536996 2986 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.018s sys 0.025s
W20260806 03:43:17.537660 1625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.282s user 0.019s sys 0.073s
I20260806 03:43:17.538470 3215 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.539048 3215 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.541042 3215 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:17.545821 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:17.579720 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 44343.9us
W20260806 03:43:17.584988 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.379s user 0.009s sys 0.025s
I20260806 03:43:17.535187 3035 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:17.592404 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.373s user 0.035s sys 0.097s
W20260806 03:43:17.594129 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.375s user 0.035s sys 0.097s
W20260806 03:43:17.587765 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.382s user 0.009s sys 0.026s
I20260806 03:43:17.486672 3107 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.599087 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:17.596746 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.42372e+06us
W20260806 03:43:17.612334 663 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.003s sys 0.022s
W20260806 03:43:17.613271 663 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.004s sys 0.022s
I20260806 03:43:17.619041 1664 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:17.620714 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:17.622270 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:17.624074 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.629905 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:17.582937 2969 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.631546 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.631522 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:17.632886 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.015s sys 0.009s
I20260806 03:43:17.633249 547 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
W20260806 03:43:17.634397 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.015s sys 0.010s
I20260806 03:43:17.628774 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.635363 776 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
W20260806 03:43:17.594715 3048 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.104s user 0.005s sys 0.022s
W20260806 03:43:17.586666 2809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.004s sys 0.018s
I20260806 03:43:17.594569 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.635108 501 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:17.639065 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.639489 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.639101 544 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:17.534567 3220 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.640381 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.663692 776 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:17.664619 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.642051 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.648636 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.653553 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.656575 547 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
W20260806 03:43:17.640511 3048 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.150s user 0.005s sys 0.022s
W20260806 03:43:17.650851 2809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.005s sys 0.019s
I20260806 03:43:17.657511 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.640818 3105 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:17.650873 3005 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.625730 3030 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.651787 2969 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.663518 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.611137 2997 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.666316 771 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:17.667224 768 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:17.667984 562 raft_consensus.cc:2379] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:17.695366 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.696182 544 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:17.696556 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.667577 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.657361 618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.249s user 0.023s sys 0.005s
W20260806 03:43:17.697588 618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.423s user 0.023s sys 0.005s
I20260806 03:43:17.666273 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:17.652980 542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.012s sys 0.016s
I20260806 03:43:17.665609 561 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:17.627926 3035 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:17.636454 3107 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:17.635895 3029 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
W20260806 03:43:17.678689 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 43690.2us
I20260806 03:43:17.682401 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.722332 767 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:17.636287 3017 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.682802 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 282) took 1497 ms. Trace:
I20260806 03:43:17.683405 669 raft_consensus.cc:595] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:17.686146 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:17.700773 661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1561) took 2156 ms (client timeout 1857 ms). Trace:
I20260806 03:43:17.636538 3068 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 03:43:17.674801 3149 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:17.706753 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:17.677831 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:17.688323 3187 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:17.720104 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.704705 1622 connection.cc:232] RPC call timeout handler was delayed by -1.1561106329318136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3149s for a response
I20260806 03:43:17.723501 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.706265 542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s user 0.012s sys 0.016s
W20260806 03:43:17.700642 1662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.003s sys 0.021s
I20260806 03:43:17.792181 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:17.702557 2836 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.793681 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:17.784529 775 rpcz_store.cc:276] 0806 03:43:16.185840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.185992 (+ 152us) service_pool.cc:227] Handling call
0806 03:43:16.186352 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:16.186598 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:43:17.188493 (+1001895us) log.cc:842] Serialized 14 byte log entry
0806 03:43:17.682308 (+493815us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:17.682404 (+ 96us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:17.682470 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.682585 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.682730 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10164,"prepare.queue_time_us":1326376,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":1731,"raft.queue_time_us":3894,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5020,"thread_start_us":1459463,"threads_started":2}
I20260806 03:43:17.691481 3105 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.784885 770 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:17.707144 2707 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.005s sys 0.018s
I20260806 03:43:17.802142 562 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:17.823787 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:17.787467 2936 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:17.802508 3017 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:17.805733 1662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s user 0.003s sys 0.021s
I20260806 03:43:17.706288 2809 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:17.714164 3224 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:17.794234 661 rpcz_store.cc:269] 0806 03:43:15.544275 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.544573 (+ 298us) service_pool.cc:227] Handling call
0806 03:43:17.700569 (+2155996us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540015c2d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:17.700703 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4916397824}
I20260806 03:43:17.907155 773 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:17.694059 3030 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.714272 3134 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:17.873431 3195 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:17.860378 3098 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:17.702008 2997 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.867805 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.873227 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 166678us
I20260806 03:43:17.871809 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.873799 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 71553.8us
W20260806 03:43:17.879971 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56308.3us
I20260806 03:43:17.714105 2833 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:17.876698 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 211) took 1818 ms. Trace:
I20260806 03:43:17.878718 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53596 (request call id 7) took 3228 ms. Trace:
I20260806 03:43:17.716450 2987 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:17.902989 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.878484 2707 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.006s sys 0.018s
I20260806 03:43:17.857909 3156 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.856448 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:17.824705 766 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:17.918507 547 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:17.860671 3223 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.897831 3186 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:17.783972 3138 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
I20260806 03:43:17.709477 3230 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.861001 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:17.948835 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 155287us
I20260806 03:43:17.980103 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:17.981276 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:17.952991 546 rpcz_store.cc:276] 0806 03:43:16.058212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.059433 (+ 1221us) service_pool.cc:227] Handling call
0806 03:43:16.182407 (+122974us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:16.216470 (+ 34063us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.216499 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:16.225328 (+ 8829us) log.cc:842] Serialized 26 byte log entry
0806 03:43:17.517329 (+1292001us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:17.517375 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.517456 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:17.876016 (+358560us) raft_consensus.cc:1671] finished
0806 03:43:17.876095 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:17.876545 (+ 450us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36395,"raft.queue_time_us":54843,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":22745,"thread_start_us":1185482,"threads_started":1,"wal-append.queue_time_us":1534058}
I20260806 03:43:17.954180 669 rpcz_store.cc:276] 0806 03:43:14.628616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.629012 (+ 396us) service_pool.cc:227] Handling call
0806 03:43:17.856725 (+3227713us) spinlock_profiling.cc:243] Waited 3.05 s on lock 0x7b540015c2d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:17.856818 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7007472512}
I20260806 03:43:17.799141 3234 consensus_queue.cc:1046] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.157s
I20260806 03:43:17.964159 498 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:17.866530 2836 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.715801 2746 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:17.721541 3185 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:17.970948 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.902465 2936 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.034150 544 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
W20260806 03:43:17.980311 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 393296us
I20260806 03:43:17.916290 3224 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.794559 3149 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.986395 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.959693 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.056392 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:17.973327 2987 raft_consensus.cc:686] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:43:18.014091 2872 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.580s user 0.011s sys 0.000s
W20260806 03:43:18.030630 3244 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260806 03:43:17.913666 2809 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:18.031474 3239 log.cc:924] Time spent T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 03:43:18.031940 2836 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.032749 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 563) took 1011 ms. Trace:
W20260806 03:43:18.036636 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 72611.8us
I20260806 03:43:18.131708 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:17.995601 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.040647 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.052134 562 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
W20260806 03:43:18.053007 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 70772.4us
I20260806 03:43:18.140625 616 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:17.941763 3134 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.958308 3098 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:18.015220 773 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:18.099110 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.034837 3048 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:17.940752 3035 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:17.967010 2833 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.015300 3229 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:43:18.060514 3092 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.792s user 0.014s sys 0.002s
I20260806 03:43:17.993191 3220 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.034421 3005 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.055751 3186 raft_consensus.cc:3032] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:18.119416 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.051946 3004 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:18.149428 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.002281 3230 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.098192 2969 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.110049 2872 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.123816 766 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:17.985483 3223 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.032891 2746 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:17.985698 3156 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.102697 548 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:18.059265 3185 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.159390 3138 consensus_queue.cc:1046] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260806 03:43:18.159816 2997 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.160166 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.160262 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.204459 671 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:18.161671 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 177) took 2547 ms. Trace:
I20260806 03:43:18.205543 663 rpcz_store.cc:276] 0806 03:43:15.614720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:16.838030 (+1223310us) service_pool.cc:227] Handling call
0806 03:43:16.838515 (+ 485us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:16.838929 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:16.838978 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:16.845675 (+ 6697us) log.cc:842] Serialized 26 byte log entry
0806 03:43:17.614269 (+768594us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:17.614315 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.615041 (+ 726us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:18.161449 (+546408us) raft_consensus.cc:1671] finished
0806 03:43:18.161468 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.161615 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4687,"raft.queue_time_us":59936,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":27226,"thread_start_us":722400,"threads_started":1,"wal-append.queue_time_us":1071529}
I20260806 03:43:18.161824 3224 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.168357 1670 raft_consensus.cc:2396] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:18.170064 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.174135 1664 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:18.190306 403 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51378 (request call id 177) took 6682 ms. Trace:
I20260806 03:43:18.199532 3185 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.198464 669 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.178328 3031 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260806 03:43:18.243451 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.159854 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.244208 767 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:18.172582 3134 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.182650 3105 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:18.201262 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:18.183032 3017 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.245997 770 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.185379 3223 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.186900 2746 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.247012 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.126067 1662 rpcz_store.cc:276] 0806 03:43:17.021739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.023688 (+ 1949us) service_pool.cc:227] Handling call
0806 03:43:17.085801 (+ 62113us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:17.130510 (+ 44709us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:17.130573 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:17.131497 (+ 924us) log.cc:842] Serialized 26 byte log entry
0806 03:43:17.913819 (+782322us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:17.922128 (+ 8309us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.941432 (+ 19304us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:18.032516 (+ 91084us) raft_consensus.cc:1671] finished
0806 03:43:18.032547 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.032683 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"thread_start_us":770476,"threads_started":1,"wal-append.queue_time_us":898148}
I20260806 03:43:18.189316 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.230350 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.248350 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:18.200781 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.248764 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:18.197556 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48236.1us
I20260806 03:43:18.249742 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.249861 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.164369 2987 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.189441 3174 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:18.191663 3195 raft_consensus.cc:3032] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:18.252009 616 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.252552 3195 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:18.193251 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.188441 3156 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:18.254230 727 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.013s sys 0.024s
W20260806 03:43:18.264038 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48582.7us
I20260806 03:43:18.266601 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.249353 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:18.274326 727 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.614s user 0.013s sys 0.024s
W20260806 03:43:18.275363 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 615436us
I20260806 03:43:18.276536 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.217482 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.274053 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.216987 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.163316 3149 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:18.170331 3004 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:18.280800 3004 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.198859 3098 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:18.282415 3195 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.254496 3159 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:18.181456 3030 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.284386 3223 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:18.284008 3159 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.173959 2872 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.172806 3230 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.286721 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.287410 776 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.168780 3186 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:18.159772 2648 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260806 03:43:18.214098 403 rpcz_store.cc:276] 0806 03:43:11.508293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:11.508540 (+ 247us) service_pool.cc:227] Handling call
0806 03:43:18.190225 (+6681685us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:13.696089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:13.696635 (+ 546us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:13.696666 (+ 31us) write_op.cc:435] Acquired schema lock
0806 03:43:13.696684 (+ 18us) tablet.cc:660] Decoding operations
0806 03:43:13.697258 (+ 574us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:13.697337 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:13.697366 (+ 29us) tablet.cc:683] Acquiring locks for 3 operations
0806 03:43:13.697712 (+ 346us) tablet.cc:699] Row locks acquired
0806 03:43:13.697733 (+ 21us) write_op.cc:260] PREPARE: finished
0806 03:43:13.697870 (+ 137us) write_op.cc:270] Start()
0806 03:43:13.697977 (+ 107us) write_op.cc:276] Timestamp: P: 1785987793697825 usec, L: 0
0806 03:43:13.697999 (+ 22us) op_driver.cc:331] REPLICATION: starting
0806 03:43:13.726947 (+ 28948us) log.cc:842] Serialized 1302 byte log entry
0806 03:43:16.014137 (+2287190us) op_driver.cc:444] REPLICATION: finished
0806 03:43:17.224565 (+1210428us) write_op.cc:301] APPLY: starting
0806 03:43:17.229552 (+ 4987us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:17.269247 (+ 39695us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:17.292191 (+ 22944us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:17.297367 (+ 5176us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:17.297684 (+ 317us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0806 03:43:17.297804 (+ 120us) write_op.cc:312] APPLY: finished
0806 03:43:17.400130 (+102326us) log.cc:842] Serialized 35 byte log entry
0806 03:43:17.922943 (+522813us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:17.943345 (+ 20402us) write_op.cc:454] Released schema lock
0806 03:43:17.945611 (+ 2266us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:17.985121 (+ 39510us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:17.994828 (+ 9707us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":1184001,"apply.run_cpu_time_us":69379,"apply.run_wall_time_us":812178,"num_ops":3,"prepare.queue_time_us":756601,"prepare.run_cpu_time_us":36487,"prepare.run_wall_time_us":557818,"raft.queue_time_us":486778,"raft.run_cpu_time_us":185628,"raft.run_wall_time_us":923354,"replication_time_us":2306337,"thread_start_us":2915353,"threads_started":5,"wal-append.queue_time_us":638661,"wal-append.run_cpu_time_us":56126,"wal-append.run_wall_time_us":2470780}]]}
I20260806 03:43:18.253866 544 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:18.182107 766 raft_consensus.cc:595] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.302600 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.303468 659 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
W20260806 03:43:18.244756 3251 log.cc:924] Time spent T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.085s user 0.000s sys 0.002s
W20260806 03:43:18.167667 3247 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260806 03:43:18.308452 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56537.6us
I20260806 03:43:18.309299 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:18.309763 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.284093 3195 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.311618 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:18.162649 2936 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.166523 2833 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.179749 2809 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.317610 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.324360 663 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:18.327636 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.328913 659 raft_consensus.cc:2396] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:18.330688 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.331782 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.168887 3187 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.147s
I20260806 03:43:18.319415 773 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:18.191895 401 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 2 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:18.336264 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:18.337090 665 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:18.337394 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.338799 401 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:18.316598 3185 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:18.341424 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:18.337302 1668 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.012s sys 0.020s
I20260806 03:43:18.342743 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.331272 2833 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:18.348098 616 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:18.325675 3174 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:18.320346 3223 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:18.362490 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.363193 659 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.364456 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:18.362609 3029 raft_consensus.cc:686] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:18.331418 3146 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:18.351652 1668 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.013s sys 0.020s
I20260806 03:43:18.342257 2809 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:18.368288 664 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.000s sys 0.067s
I20260806 03:43:18.335785 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.384033 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 479) took 1041 ms. Trace:
I20260806 03:43:18.389582 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:18.392817 664 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.599s user 0.000s sys 0.068s
I20260806 03:43:18.388667 616 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:18.394747 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:18.395952 3256 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 03:43:18.396080 548 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:18.392314 1668 rpcz_store.cc:276] 0806 03:43:17.343098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.343246 (+ 148us) service_pool.cc:227] Handling call
0806 03:43:17.374060 (+ 30814us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.374320 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:17.787147 (+412827us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.383697 (+596550us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:18.383747 (+ 50us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:18.383789 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:18.383852 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.383978 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":630239,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":96,"spinlock_wait_cycles":11136,"thread_start_us":952765,"threads_started":2}
I20260806 03:43:18.350806 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.396669 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.389657 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.397394 558 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:18.397468 562 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:18.398669 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.398818 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.400449 773 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:18.383599 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.016s sys 0.054s
W20260806 03:43:18.401707 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s user 0.016s sys 0.054s
I20260806 03:43:18.401877 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:18.404932 2936 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:18.404726 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:18.405407 2936 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:18.405831 2936 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:18.358548 3174 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.406849 3174 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.409781 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.411127 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.411748 546 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:18.413589 2936 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.415637 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.416193 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.417088 3187 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:18.356379 3185 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.386520 3233 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:18.422546 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 569) took 1015 ms. Trace:
I20260806 03:43:18.427109 616 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:18.425513 1670 rpcz_store.cc:276] 0806 03:43:17.407483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.407622 (+ 139us) service_pool.cc:227] Handling call
0806 03:43:18.420400 (+1012778us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b540067e154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:18.422467 (+ 2067us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1702499072}
I20260806 03:43:18.428900 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.429620 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2019) took 2235 ms. Trace:
I20260806 03:43:18.366674 401 catalog_manager.cc:5495] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65), VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66), VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67), VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: NON_VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:18.430136 664 rpcz_store.cc:276] 0806 03:43:16.194531 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.181520 (+986989us) service_pool.cc:227] Handling call
0806 03:43:17.185933 (+ 4413us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:17.186360 (+ 427us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:17.186419 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:17.186965 (+ 546us) log.cc:842] Serialized 26 byte log entry
0806 03:43:17.707408 (+520443us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:18.244460 (+537052us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b24001d5c28. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f321324fcf4 00007f321324ddfb 00007f32132dc559 00007f32132dc1e5 00007f32132db82d 00007f3215f8a46e 00007f3215fa1c0c 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551
0806 03:43:18.393274 (+148814us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.419845 (+ 26571us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:18.419939 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:18.419993 (+ 54us) raft_consensus.cc:1671] finished
0806 03:43:18.428423 (+ 8430us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.429538 (+ 1115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33923,"raft.queue_time_us":30486,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":11302,"spinlock_wait_cycles":81301760,"thread_start_us":1065018,"threads_started":2,"wal-append.queue_time_us":830666}
I20260806 03:43:18.433624 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.434718 773 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.441412 401 catalog_manager.cc:5495] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 3, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:18.384891 2746 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:18.387996 3149 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.442967 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:18.445883 1622 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.007s sys 0.023s
W20260806 03:43:18.473922 1622 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.561s user 0.007s sys 0.023s
I20260806 03:43:18.473789 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:18.474298 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 562082us
I20260806 03:43:18.476363 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.477131 669 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:18.441056 1665 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.010s sys 0.010s
W20260806 03:43:18.421758 3026 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.023s sys 0.030s
I20260806 03:43:18.384616 3030 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:18.427242 3185 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:18.457964 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.494320 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.429834 2833 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:18.526883 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.528841 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.530838 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:18.479890 1665 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.011s sys 0.010s
W20260806 03:43:18.493093 3026 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.621s user 0.024s sys 0.030s
I20260806 03:43:18.419190 3146 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.481500 3232 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:18.458958 2746 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.542279 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.545192 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:18.551755 1671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.003s sys 0.052s
W20260806 03:43:18.569491 1671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s user 0.004s sys 0.052s
I20260806 03:43:18.440912 3233 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.556231 665 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.545094 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.526563 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:18.569988 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.572634 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:18.526521 2833 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.570418 769 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:18.539155 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.581924 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:18.540745 3206 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.105s user 0.013s sys 0.003s
I20260806 03:43:18.566642 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 985) took 3769 ms. Trace:
I20260806 03:43:18.584085 2833 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:18.583971 3004 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:18.569798 2707 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.050s sys 0.034s
I20260806 03:43:18.592307 3146 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:18.596859 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.526943 3030 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:18.603135 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 287889us
I20260806 03:43:18.597853 542 rpcz_store.cc:276] 0806 03:43:14.774486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:14.790811 (+ 16325us) service_pool.cc:227] Handling call
0806 03:43:14.813607 (+ 22796us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:14.939429 (+125822us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:14.939472 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:17.108095 (+2168623us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b2c00056a28. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:43:17.109586 (+ 1491us) log.cc:842] Serialized 590 byte log entry
0806 03:43:17.858239 (+748653us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:17.879154 (+ 20915us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:17.895911 (+ 16757us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:18.495606 (+599695us) raft_consensus.cc:1671] finished
0806 03:43:18.497510 (+ 1904us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.543358 (+ 45848us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":124887,"raft.queue_time_us":125814,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":659151232,"thread_start_us":681662,"threads_started":1,"wal-append.queue_time_us":849486}
I20260806 03:43:18.611910 3030 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.612192 3004 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.612929 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.614588 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.614940 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:18.615489 3098 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:18.542277 3232 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.570519 2746 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:18.529906 3149 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:18.570568 3178 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.011s sys 0.048s
W20260806 03:43:18.606583 2707 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.051s sys 0.035s
I20260806 03:43:18.564978 3029 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.621119 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.621997 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.646875 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.649442 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.660746 775 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:18.661686 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.661926 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.662549 669 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:18.651618 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:18.652251 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 331686us
I20260806 03:43:18.653717 771 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:18.663344 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.660245 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.654696 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.657791 1668 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:18.627604 3232 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:18.673748 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.675101 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:18.655427 3178 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.935s user 0.011s sys 0.049s
I20260806 03:43:18.678222 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.702739 564 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:18.667249 3098 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:18.709349 3174 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.714440 1658 raft_consensus.cc:1226] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:18.718729 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.720337 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.721031 546 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:18.723413 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:18.736483 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.736521 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.738301 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.679003 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 260) took 1413 ms. Trace:
I20260806 03:43:18.731153 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.729200 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.743101 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.723418 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.746167 776 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:18.747426 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.748611 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:18.754583 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:18.754786 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:18.756362 558 raft_consensus.cc:2362] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3. Re-sending same reply.
I20260806 03:43:18.756336 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:18.759421 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:18.729897 3253 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 03:43:18.756381 546 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:18.762226 614 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:18.724107 3174 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.728332 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.728223 3149 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:18.736308 770 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:18.772141 616 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.807554 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:18.774600 775 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:18.775059 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.808717 775 raft_consensus.cc:595] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.809068 775 raft_consensus.cc:603] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:18.811062 773 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:18.766325 3105 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:18.762815 3223 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:18.722800 3265 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260806 03:43:18.767716 3068 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:18.813190 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:18.725652 3185 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.771659 3236 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:18.824365 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:18.821514 772 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:18.811954 3174 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:18.813377 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 234224us
I20260806 03:43:18.811764 766 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:18.768246 3146 consensus_queue.cc:1046] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260806 03:43:18.800621 3233 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.812484 3098 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:18.800272 3107 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260806 03:43:18.822166 3068 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:18.876883 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.812304 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 261) took 1374 ms. Trace:
I20260806 03:43:18.873458 768 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba"
dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:58492
I20260806 03:43:18.885885 768 raft_consensus.cc:595] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:18.878296 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.827889 3236 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:18.881428 3149 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:43:18.882544 3278 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260806 03:43:18.901518 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.923281 1664 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:18.844429 3181 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:18.924397 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.833174 3174 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.885593 1671 rpcz_store.cc:276] 0806 03:43:17.438358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.478471 (+ 40113us) service_pool.cc:227] Handling call
0806 03:43:17.489015 (+ 10544us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.489217 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:17.916731 (+427514us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.811614 (+894883us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.811901 (+ 287us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:18.811960 (+ 59us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:18.812029 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:18.812107 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.812232 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9988,"prepare.queue_time_us":541405,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":84,"raft.queue_time_us":8413,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":22471,"spinlock_wait_cycles":14500224,"thread_start_us":1299390,"threads_started":3,"wal-append.queue_time_us":761533}
I20260806 03:43:18.847543 3098 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:18.830112 3223 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260806 03:43:18.842873 3284 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:18.905929 3262 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260806 03:43:18.930015 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:43:18.943729 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:18.879774 3068 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:18.933692 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 90438.8us
I20260806 03:43:18.918427 3236 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.937844 3105 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:18.944674 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:18.929606 3174 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:18.918350 3149 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:18.931118 3098 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:18.954821 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.955652 2746 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.952946 3105 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:18.970113 727 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:18.974838 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:18.934413 3284 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:18.910054 3275 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20260806 03:43:18.982398 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:18.968225 3149 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:18.983349 563 raft_consensus.cc:1226] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:18.985647 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:18.978855 3273 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.243s user 0.001s sys 0.001s
I20260806 03:43:18.973304 2707 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:18.990684 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:18.978605 2836 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:18.996541 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:18.983822 3107 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:18.933779 3181 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:18.993742 773 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:18.979521 3284 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:19.035107 727 connection.cc:232] RPC call timeout handler was delayed by -1.34822429693304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
I20260806 03:43:19.000788 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:19.037953 727 connection.cc:232] RPC call timeout handler was delayed by -1.24022429715842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
I20260806 03:43:19.039619 3225 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:19.040329 3225 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:19.043541 3181 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.049278 3105 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:19.058660 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1656) took 1436 ms. Trace:
I20260806 03:43:19.059237 564 rpcz_store.cc:276] 0806 03:43:17.621834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.621966 (+ 132us) service_pool.cc:227] Handling call
0806 03:43:19.058411 (+1436445us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b540011e9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:19.058571 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2245712256}
W20260806 03:43:19.039620 727 connection.cc:232] RPC call timeout handler was delayed by -1.24022429715842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.192s for a response
W20260806 03:43:19.062469 727 connection.cc:232] RPC call timeout handler was delayed by -1.1034242969471961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
W20260806 03:43:19.062775 727 connection.cc:232] RPC call timeout handler was delayed by -1.046724297106266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2135s for a response
I20260806 03:43:18.985826 3185 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.745252 1665 rpcz_store.cc:276] 0806 03:43:17.265823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.269737 (+ 3914us) service_pool.cc:227] Handling call
0806 03:43:17.336008 (+ 66271us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:17.336358 (+ 350us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:43:17.877463 (+541105us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.607075 (+729612us) log.cc:842] Serialized 14 byte log entry
0806 03:43:18.619609 (+ 12534us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:18.639345 (+ 19736us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:18.649962 (+ 10617us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:18.659608 (+ 9646us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:18.675425 (+ 15817us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19436,"prepare.queue_time_us":26706,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":24720,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":5409,"thread_start_us":660153,"threads_started":1,"wal-append.queue_time_us":789941}
W20260806 03:43:19.063032 727 connection.cc:232] RPC call timeout handler was delayed by -1.0269242969807237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
I20260806 03:43:19.066282 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.067307 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.067482 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.067687 3187 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.068830 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:19.017805 2746 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:18.995584 3094 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.069788 3094 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.070423 3232 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.071646 3282 raft_consensus.cc:3032] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:18.993472 2707 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:19.072036 3282 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:19.072237 2707 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:19.016741 3159 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:19.073014 3282 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.073522 2936 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.009518 3186 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.074739 3283 raft_consensus.cc:3032] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:19.075146 3283 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:19.075198 3186 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:19.076002 3283 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.077337 3283 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.023605 1662 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:19.074230 3282 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.028317 3174 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.078763 3156 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:19.036114 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 428) took 1510 ms. Trace:
W20260806 03:43:19.095692 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 62998.5us
I20260806 03:43:19.095366 667 rpcz_store.cc:276] 0806 03:43:17.520699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.521902 (+ 1203us) service_pool.cc:227] Handling call
0806 03:43:19.029992 (+1508090us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321b264c6e 00007f321495b807 00007f321495b792 00007f321495b6c1 00007f321495b63c 00007f321495a8f2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:19.030525 (+ 533us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2978153856}
W20260806 03:43:19.063011 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 62322.4us
I20260806 03:43:19.099002 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.068747 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.030342 3107 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:19.030699 1671 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:19.083813 1622 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.085680 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1655) took 1461 ms. Trace:
I20260806 03:43:19.088414 671 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:19.090757 668 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:19.091958 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.118687 669 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:19.092578 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.094309 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.120754 674 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:19.116155 563 rpcz_store.cc:276] 0806 03:43:17.621378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:17.621493 (+ 115us) service_pool.cc:227] Handling call
0806 03:43:17.638060 (+ 16567us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:18.986125 (+1348065us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:19.082085 (+ 95960us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54001784d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:19.082314 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3093485312}
W20260806 03:43:18.997428 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 181116us
I20260806 03:43:19.101441 659 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:19.126864 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.132638 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:18.972219 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 210088us
I20260806 03:43:19.134609 614 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.152809 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.154165 773 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:19.041489 3225 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.156755 615 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.072451 3232 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.072041 3094 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:19.129415 3107 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.113345 2700 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:19.095175 3156 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:19.074249 3146 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:19.136057 2746 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.168248 2936 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:19.177374 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 108599us
I20260806 03:43:19.036022 2833 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.185555 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.186712 615 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:19.150616 3267 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:19.189143 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 94420.5us
I20260806 03:43:19.190043 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.175091 3094 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.195101 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:19.192083 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 63038us
I20260806 03:43:19.177640 2746 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:19.177870 2936 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:19.195519 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61034.6us
I20260806 03:43:19.200862 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.205782 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.207535 3068 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.209131 2833 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.209614 3068 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.196053 3094 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:19.210407 3094 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.187700 3185 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.191136 3134 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.216710 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.193347 3232 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:19.222712 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.192293 3301 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.223527 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:19.224005 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.211023 2833 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:19.225593 3035 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:19.225683 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 141993us
I20260806 03:43:19.226853 659 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:19.213033 2746 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.190510 3146 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260806 03:43:19.229178 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.230144 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.225998 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.232470 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.212345 2936 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.236718 1622 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.220320 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:19.236688 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 58156.8us
I20260806 03:43:19.193625 3098 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:19.311791 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 91590.1us
I20260806 03:43:19.213739 773 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:19.314713 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.214783 3313 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:19.313090 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 87164.6us
I20260806 03:43:19.238781 3035 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:19.323161 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.202920 3295 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.213201 3243 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20260806 03:43:19.325780 614 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:19.326213 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 93813.3us
I20260806 03:43:19.340903 663 raft_consensus.cc:1226] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:19.327535 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.239423 3195 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.223467 1665 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:19.222563 3234 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.233062 3232 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:19.343662 3232 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:19.236682 3220 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:19.346576 3220 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.346967 3220 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:19.215653 3284 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.351809 3284 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:19.329332 3031 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.352398 3243 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:19.352948 3243 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:19.353214 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.354444 498 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.355542 3267 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:19.356184 3267 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.356664 3267 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:19.357215 3267 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.358402 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.359048 3234 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.359754 3234 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:19.360392 3234 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.360543 3236 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.324399 3174 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.334892 3035 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.334481 3068 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:19.326843 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 90211us
I20260806 03:43:19.342038 501 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.367779 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.368114 3236 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:19.368609 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:19.369154 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.342695 3295 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.343015 3195 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.323464 3223 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.340801 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 1945) took 1009 ms. Trace:
I20260806 03:43:19.383574 769 rpcz_store.cc:276] 0806 03:43:18.331440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:18.331559 (+ 119us) service_pool.cc:227] Handling call
0806 03:43:19.338267 (+1006708us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54002bd9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:19.340744 (+ 2477us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":547079040}
W20260806 03:43:19.370472 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 47340.3us
I20260806 03:43:19.385248 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.370350 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.324630 3206 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.328843 3187 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.427255 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:19.428614 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.428982 544 raft_consensus.cc:2396] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:19.374101 3068 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.415517 3206 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:19.211660 3156 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.430768 3206 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.383039 2911 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.433241 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.369864 3305 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.234929 3149 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.448637 727 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.415522 2648 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.387041 3195 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.386960 3319 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:19.415220 3265 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.354041 3243 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.432325 3146 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.383122 3313 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:19.442865 3283 raft_consensus.cc:2741] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 03:43:19.467334 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.467622 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.450011 727 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.469970 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.480937 661 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:19.446610 3284 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.383733 2700 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:19.457842 3236 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.484726 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.456426 3322 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.491852 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:19.499023 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 155479us
I20260806 03:43:19.434527 3275 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 03:43:19.475138 3319 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:19.501825 3236 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.502007 614 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.504452 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.489945 3283 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.506665 614 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:19.507570 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.509500 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.511556 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.512219 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.490702 3146 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:19.514557 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.516228 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.512378 664 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:19.520182 659 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:19.520417 667 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:19.520579 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.470898 3233 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260806 03:43:19.526501 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.533876 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.515534 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.520833 2833 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.559600 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.562330 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.529156 3230 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.531497 3187 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:19.526644 3178 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:19.615823 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 261456us
I20260806 03:43:19.548849 3146 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.549873 3223 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:19.610527 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.532671 3098 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.594321 776 raft_consensus.cc:2354] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:19.544081 3283 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:19.585947 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 216239us
I20260806 03:43:19.630601 501 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:19.551762 3328 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.633294 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:19.634089 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 105491us
I20260806 03:43:19.581861 3105 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:19.625726 664 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:19.574131 2833 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:19.623759 2707 consensus_meta.cc:287] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.551s user 0.017s sys 0.000s
I20260806 03:43:19.617051 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.616837 3149 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:19.652694 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 68919.6us
I20260806 03:43:19.552431 2936 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.639742 3029 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.651955 498 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.671720 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.666884 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.668045 616 leader_election.cc:398] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:19.673454 616 leader_election.cc:401] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:43:19.636674 3223 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:19.675345 769 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:19.675771 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.644948 3305 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:19.649111 2833 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.573342 2700 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.584779 3107 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.684975 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.686299 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.613548 3332 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260806 03:43:19.626740 3187 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.686314 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.674777 3029 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.583546 3313 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.670646 3230 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:19.693056 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.683897 3098 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:19.709951 3220 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.714022 3220 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.674757 3336 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:19.702855 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 67082.7us
I20260806 03:43:19.652420 2707 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.688570 3265 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:19.717036 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:19.699033 3105 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.728395 664 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:19.730033 668 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:19.703612 3031 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.703846 3029 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.734148 3329 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:19.734720 3265 raft_consensus.cc:686] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:19.734489 3223 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.735809 2707 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.739650 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.690627 3226 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 03:43:19.718106 3159 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:19.745846 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:19.746845 3094 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.536s user 0.013s sys 0.000s
I20260806 03:43:19.702354 3186 consensus_queue.cc:1046] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260806 03:43:19.747680 2746 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.729655 3105 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.767418 3105 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.738087 3031 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:19.748555 3329 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.744206 2836 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:19.791146 2836 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:19.791585 2836 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:19.724074 3336 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:19.792148 3336 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:19.645330 3319 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:19.754715 3314 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.217s user 0.000s sys 0.001s
I20260806 03:43:19.755668 3187 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.729332 3328 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:19.688544 3305 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.721756 3230 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.766840 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:19.701794 3295 consensus_queue.cc:1046] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 03:43:19.795528 665 raft_consensus.cc:1226] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:19.763494 3149 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.790189 3031 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:19.805528 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 131299us
I20260806 03:43:19.821331 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:19.829790 3305 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.834095 2746 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.803203 3328 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.783833 3094 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:19.838908 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 187046us
I20260806 03:43:19.840230 498 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:19.844573 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.839535 3328 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:19.831650 3187 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:19.804325 3206 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.850801 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.824209 2936 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.853997 674 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:19.854439 3035 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.856127 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.859871 1664 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:19.864485 3275 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:19.869112 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:19.871624 674 raft_consensus.cc:1226] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:19.858690 3187 raft_consensus.cc:686] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:19.883807 3187 consensus_queue.cc:237] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.859300 3334 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:19.888942 659 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:19.891283 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:19.890467 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.886204 3275 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.875423 3107 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.905460 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1127) took 1177 ms. Trace:
W20260806 03:43:19.890210 3234 consensus_meta.cc:287] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.505s user 0.015s sys 0.000s
W20260806 03:43:19.890877 3232 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.547s user 0.009s sys 0.003s
I20260806 03:43:19.879487 3357 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:19.835563 3265 consensus_queue.cc:237] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.912688 772 rpcz_store.cc:276] 0806 03:43:18.728248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:18.728387 (+ 139us) service_pool.cc:227] Handling call
0806 03:43:18.761954 (+ 33567us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:18.821389 (+ 59435us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:43:18.892491 (+ 71102us) log.cc:842] Serialized 14 byte log entry
0806 03:43:19.124833 (+232342us) log.cc:842] Serialized 14 byte log entry
0806 03:43:19.617530 (+492697us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:19.617601 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:19.617654 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:19.905280 (+287626us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b5400345654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:19.905318 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:19.905410 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40916,"prepare.queue_time_us":193207,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":116210,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":24470,"spinlock_wait_cycles":136155520,"thread_start_us":679232,"threads_started":3,"wal-append.queue_time_us":264824}
I20260806 03:43:19.922210 3232 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:19.946230 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 261806us
I20260806 03:43:19.952767 772 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:19.953881 669 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:19.953501 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:19.957417 3275 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.958083 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:19.959007 3329 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.931835 3234 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:19.964053 3035 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.964260 3234 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.933027 3357 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:19.966639 3232 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:19.967374 3357 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.967886 3094 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:19.979962 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:19.983038 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:19.986102 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:19.987289 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:19.987211 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:19.996883 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:19.987318 1663 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:19.974406 3289 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:20.007246 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.005851 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.012552 769 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 2: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:19.986099 3206 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.016155 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.019441 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:19.995788 3230 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.020329 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.025766 561 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.021951 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.023774 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.024219 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.026763 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.004117 1657 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:20.028780 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.030838 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.035638 547 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.036198 564 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:20.037716 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.038329 561 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
W20260806 03:43:20.042073 3267 consensus_meta.cc:287] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.684s user 0.006s sys 0.011s
I20260806 03:43:20.042713 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.042582 3267 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.044719 558 raft_consensus.cc:1226] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.046252 561 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.013705 3289 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:20.036324 562 raft_consensus.cc:2379] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.047344 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.049312 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.018975 3233 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:20.013908 3234 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:20.051003 562 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.035257 3206 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:20.009127 2836 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.055542 564 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:20.058928 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.020212 3223 consensus_queue.cc:1046] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 03:43:20.067296 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.068293 562 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:20.069974 564 raft_consensus.cc:1226] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.076568 547 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:20.081306 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.081851 547 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:20.079356 562 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:20.083439 3289 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.087234 770 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:20.056666 3233 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.088274 2836 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:20.070554 3230 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:20.089726 3233 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.089363 3230 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.091686 561 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:20.092752 564 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
W20260806 03:43:20.072616 3195 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.583s user 0.014s sys 0.000s
I20260806 03:43:20.097869 3275 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:20.113860 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.113519 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.115461 674 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.115514 562 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.116509 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.059815 3206 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.120476 773 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:20.058524 3234 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.100612 3365 raft_consensus.cc:3032] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
W20260806 03:43:20.132434 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 117265us
I20260806 03:43:20.133001 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.103222 3267 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.138104 727 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.140106 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.142000 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:20.124361 3226 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:20.142920 3305 consensus_queue.cc:1046] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:20.143014 3226 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.150144 674 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:20.145788 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:20.151829 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.152112 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:20.152581 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.150640 766 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:20.114709 3328 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.153927 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.158020 773 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.075819 3367 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260806 03:43:20.159255 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.104291 3195 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.173352 3361 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:20.173209 2872 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:20.135058 3365 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:20.176438 3105 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.177218 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:20.178706 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.179816 498 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.196576 667 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:20.180426 772 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:20.181070 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.197311 775 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:20.182444 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.198065 776 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:20.183205 3243 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:20.181077 3267 consensus_queue.cc:1046] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:43:20.135329 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:20.185976 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.180450 3234 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.190935 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.119160 3275 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:20.199901 2833 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.507s user 0.008s sys 0.005s
I20260806 03:43:20.193427 3328 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:20.151196 3107 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:20.134017 3098 consensus_queue.cc:1046] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260806 03:43:20.215225 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.205458 775 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:43:20.215869 498 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.187072 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:20.192579 3331 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:20.217626 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.144905 3319 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:20.150983 3181 raft_consensus.cc:3032] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:20.212401 771 raft_consensus.cc:1226] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:20.229727 499 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:20.202342 3289 consensus_queue.cc:1046] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
W20260806 03:43:20.237022 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 81163.8us
I20260806 03:43:20.232299 542 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.239620 727 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:20.209443 3373 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:20.211895 3365 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.234791 3361 consensus_queue.cc:1046] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:43:20.250721 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 92880.5us
I20260806 03:43:20.242746 3105 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:20.244932 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.248348 3267 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:20.248929 3336 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.227982 3187 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.234910 3357 consensus_queue.cc:1046] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:20.231156 3181 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:20.256174 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:20.258316 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 74139.5us
I20260806 03:43:20.259614 3234 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:20.269822 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:20.260699 1657 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:20.214327 2833 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.278549 2833 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.278667 3365 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.260957 3361 raft_consensus.cc:3032] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:20.279357 3361 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.280179 3361 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.281371 3361 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.282770 546 raft_consensus.cc:2379] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.284225 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.214473 3029 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.250556 3230 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.244691 3226 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:20.274718 667 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:20.252120 3319 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:20.276613 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.278093 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.285511 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.259136 498 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.244475 3367 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.286285 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.265873 3149 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.301033 775 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:20.302799 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 43766.9us
I20260806 03:43:20.303789 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.304121 498 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.305730 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.319803 3275 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.295060 2707 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:20.295742 3187 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 03:43:20.321012 2707 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:20.321959 3187 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:20.322705 3187 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.322085 2707 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.330148 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.275122 3305 raft_consensus.cc:2741] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:43:20.339797 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.283849 3206 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.283671 3336 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.329582 3029 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.347573 773 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:20.300344 3319 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:20.325980 3159 raft_consensus.cc:3032] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:20.300426 3388 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:20.337777 3301 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:20.343444 3367 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.275939 3234 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.431399 775 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:20.344779 3195 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:20.454548 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45897us
I20260806 03:43:20.455847 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:20.324568 3031 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.404606 3319 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:20.439954 3387 log.cc:924] Time spent T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260806 03:43:20.404314 3159 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:20.349738 3226 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.105s
I20260806 03:43:20.476066 771 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:20.478242 563 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:20.475812 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 165141us
I20260806 03:43:20.403079 3236 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260806 03:43:20.458626 3388 raft_consensus.cc:686] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:20.483235 3206 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:20.497445 3206 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.498947 3206 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.502557 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:20.503525 3329 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:20.481777 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 496) took 1402 ms. Trace:
I20260806 03:43:20.517325 1624 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:20.467656 2833 consensus_queue.cc:1046] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260806 03:43:20.447378 3301 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.413340 3181 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.504164 3329 raft_consensus.cc:686] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:20.523478 3159 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.504103 3328 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260806 03:43:20.562688 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 257040us
I20260806 03:43:20.576159 562 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:20.586851 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 335177us
I20260806 03:43:20.581969 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.536048 3159 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.591950 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.521632 3195 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260806 03:43:20.599457 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.592760 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.602449 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.504889 3394 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260806 03:43:20.584986 3031 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.587512 3236 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260806 03:43:20.617595 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.617817 3234 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.619076 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.620131 564 raft_consensus.cc:2379] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.616219 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.622140 562 raft_consensus.cc:2379] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.622105 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.623508 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.627632 562 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:20.592751 3181 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.631793 3301 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.634799 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:20.620790 3328 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:20.511052 1665 rpcz_store.cc:276] 0806 03:43:19.078416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.095301 (+ 16885us) service_pool.cc:227] Handling call
0806 03:43:19.175237 (+ 79936us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:19.343136 (+167899us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:20.433676 (+1090540us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b54006d24d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:20.475424 (+ 41748us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":79831424}
I20260806 03:43:20.644062 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.640630 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:20.644850 562 raft_consensus.cc:2362] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4. Re-sending same reply.
I20260806 03:43:20.584918 3388 consensus_queue.cc:237] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.591333 2833 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:20.646015 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.652393 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.654268 2833 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.640127 3098 consensus_queue.cc:1046] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260806 03:43:20.657225 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.653407 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.664453 3178 raft_consensus.cc:3032] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:20.665437 3178 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:20.657223 3301 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:20.667989 3301 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.669286 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1600) took 1003 ms. Trace:
I20260806 03:43:20.670823 3178 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.673439 3178 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:20.669777 671 rpcz_store.cc:276] 0806 03:43:19.666326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.666532 (+ 206us) service_pool.cc:227] Handling call
0806 03:43:20.669026 (+1002494us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b540015c7d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:20.669206 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571161472}
I20260806 03:43:20.647589 3328 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:20.649863 3329 consensus_queue.cc:237] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.684108 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.689247 564 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:20.690104 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.686792 3399 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260806 03:43:20.691155 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.691874 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:20.693228 674 raft_consensus.cc:1226] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:20.695765 674 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:20.697029 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:20.691025 661 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.697834 671 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:20.664645 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.700117 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:20.703224 544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 223) took 2307 ms (client timeout 2071 ms). Trace:
W20260806 03:43:20.706009 544 rpcz_store.cc:269] 0806 03:43:18.395363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:18.477780 (+ 82417us) service_pool.cc:227] Handling call
0806 03:43:20.702941 (+2225161us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54001c89d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:20.703090 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2061827968}
I20260806 03:43:20.706930 3174 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:20.707410 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:20.707475 3174 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:20.708272 544 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:20.708783 3233 raft_consensus.cc:3032] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:20.709427 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:20.709571 3328 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.705123 3156 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:20.679693 3400 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 03:43:20.709903 3404 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:20.708822 3174 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.709444 3233 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.686916 3301 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.710110 544 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:20.720850 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:20.701005 3195 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:20.717419 3405 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.018s
I20260806 03:43:20.729254 3233 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.723291 3341 raft_consensus.cc:3032] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:20.731038 3350 raft_consensus.cc:3032] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:20.731554 3233 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:20.731978 3350 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.731127 3341 raft_consensus.cc:729] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:20.732443 3400 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:20.732105 3195 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.734454 3400 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:20.738641 3301 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:20.770030 3243 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.565s user 0.011s sys 0.003s
I20260806 03:43:20.774411 498 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:20.746124 3301 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.735931 3400 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.834975 564 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:20.807181 3187 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:20.813720 3107 consensus_meta.cc:287] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.595s user 0.011s sys 0.003s
W20260806 03:43:20.856819 3362 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.807s user 0.000s sys 0.013s
I20260806 03:43:20.748939 3341 consensus_queue.cc:260] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.859261 3341 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:20.859536 3374 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.778s user 0.000s sys 0.005s
I20260806 03:43:20.857057 3319 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.814677 3243 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:20.873240 3352 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:20.853444 3195 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.815184 3350 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.865557 3107 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:20.866031 3187 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:20.937662 547 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:20.939368 562 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:20.939591 544 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:20.888275 2833 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.917116 3334 raft_consensus.cc:3032] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:20.955456 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 208) took 1440 ms. Trace:
I20260806 03:43:20.956013 659 rpcz_store.cc:276] 0806 03:43:19.515495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.540298 (+ 24803us) service_pool.cc:227] Handling call
0806 03:43:19.547430 (+ 7132us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:19.889483 (+342053us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:20.955092 (+1065609us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b540036d8d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:20.955355 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784814336}
I20260806 03:43:20.963649 3187 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:20.975894 3319 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:20.977703 3301 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.880668 3243 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.950995 3334 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.878638 3348 consensus_queue.cc:1046] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20260806 03:43:20.993013 3357 raft_consensus.cc:3032] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:20.993644 3357 raft_consensus.cc:729] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:20.995325 3334 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:20.999951 3334 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:21.007097 3275 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.037432 3275 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.011600 3243 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:20.995110 3357 consensus_queue.cc:260] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.017407 3174 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:21.049690 3371 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.955s user 0.003s sys 0.002s
I20260806 03:43:21.052201 3223 raft_consensus.cc:3032] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:21.053001 3223 raft_consensus.cc:729] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:21.056010 3345 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 1.249s user 0.000s sys 0.004s
I20260806 03:43:21.065279 3404 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:21.070164 3223 consensus_queue.cc:260] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.072402 3223 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:21.073201 3275 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.079391 3397 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:21.080180 3397 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:21.085060 3149 raft_consensus.cc:3032] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:21.085829 3149 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:21.082206 3397 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.088318 3397 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:21.092828 3350 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:21.087363 3149 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:20.186100 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:21.189131 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1686) took 1165 ms. Trace:
I20260806 03:43:21.189852 562 raft_consensus.cc:1226] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:43:20.941890 544 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 03:43:21.448336 544 pending_rounds.cc:76] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 5
I20260806 03:43:21.448870 544 pending_rounds.cc:97] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 03:43:21.449460 544 raft_consensus.cc:2866] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:21.192778 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1685) took 1170 ms. Trace:
I20260806 03:43:21.452405 546 rpcz_store.cc:276] 0806 03:43:20.022641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.023477 (+ 836us) service_pool.cc:227] Handling call
0806 03:43:21.191930 (+1168453us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b540031f6d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:21.192493 (+ 563us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":588084480}
I20260806 03:43:21.451704 400 catalog_manager.cc:5495] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 1 to 2, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:21.454663 3440 log.cc:924] Time spent T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.096s user 0.001s sys 0.002s
I20260806 03:43:21.198058 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1683) took 1176 ms. Trace:
I20260806 03:43:21.199456 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 228) took 1152 ms. Trace:
I20260806 03:43:21.200697 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1680) took 1181 ms. Trace:
I20260806 03:43:21.204104 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 216) took 1226 ms. Trace:
I20260806 03:43:21.205632 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2051) took 2114 ms. Trace:
I20260806 03:43:21.208374 664 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 03:43:21.215456 669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 204) took 1701 ms (client timeout 2112 ms). Trace:
I20260806 03:43:21.221904 668 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:21.234889 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 497) took 1784 ms. Trace:
I20260806 03:43:21.235430 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:21.292431 1671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s user 0.021s sys 0.024s
I20260806 03:43:21.298218 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 1974) took 1292 ms. Trace:
I20260806 03:43:21.306970 542 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:21.357831 401 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35374 (request call id 274) took 5967 ms. Trace:
W20260806 03:43:21.402767 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.020s sys 0.017s
I20260806 03:43:21.421363 558 rpcz_store.cc:276] 0806 03:43:20.023004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.023138 (+ 134us) service_pool.cc:227] Handling call
0806 03:43:20.039937 (+ 16799us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:20.047637 (+ 7700us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.188279 (+1140642us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b540011f3d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.188780 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10337536}
I20260806 03:43:21.196130 547 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:21.455183 546 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260806 03:43:21.457020 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.003s sys 0.034s
I20260806 03:43:21.457389 3149 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:21.474190 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.022s sys 0.050s
W20260806 03:43:21.470618 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.871s user 0.021s sys 0.017s
W20260806 03:43:21.474542 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.032s sys 0.072s
W20260806 03:43:21.474781 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.023s sys 0.050s
W20260806 03:43:21.475206 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.009s sys 0.031s
W20260806 03:43:21.475730 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.652s user 0.010s sys 0.032s
W20260806 03:43:21.477188 1625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.821s user 0.010s sys 0.027s
W20260806 03:43:21.477685 1625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.011s sys 0.027s
W20260806 03:43:21.478497 775 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.003s sys 0.024s
W20260806 03:43:21.478986 775 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s user 0.004s sys 0.024s
W20260806 03:43:21.480357 1671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.156s user 0.026s sys 0.024s
I20260806 03:43:21.479947 400 catalog_manager.cc:5495] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 2 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:21.483090 3329 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.003s sys 0.010s
W20260806 03:43:21.483608 3329 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s user 0.003s sys 0.010s
I20260806 03:43:21.483479 3365 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:21.484921 3389 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:21.484586 3365 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:21.475085 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.971s user 0.032s sys 0.073s
W20260806 03:43:21.487016 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 982687us
W20260806 03:43:21.475050 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 875543us
I20260806 03:43:21.461369 561 rpcz_store.cc:276] 0806 03:43:20.046918 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.047121 (+ 203us) service_pool.cc:227] Handling call
0806 03:43:21.199062 (+1151941us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b540032d2d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:21.199246 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94911872}
I20260806 03:43:21.462647 1668 rpcz_store.cc:276] 0806 03:43:19.450253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.450481 (+ 228us) service_pool.cc:227] Handling call
0806 03:43:19.450875 (+ 394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:19.451244 (+ 369us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:19.451282 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:19.533234 (+ 81952us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b2c001519d8. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:43:19.557357 (+ 24123us) log.cc:842] Serialized 26 byte log entry
0806 03:43:19.682894 (+125537us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:19.682935 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:19.683006 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:21.055075 (+1372069us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:21.233997 (+178922us) raft_consensus.cc:1671] finished
0806 03:43:21.234047 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:21.234501 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12109,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":10479,"spinlock_wait_cycles":22145664,"thread_start_us":112325,"threads_started":1,"wal-append.queue_time_us":190153}
W20260806 03:43:21.491854 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.885s user 0.004s sys 0.035s
W20260806 03:43:21.478873 769 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1146) took 1460 ms (client timeout 1698 ms). Trace:
I20260806 03:43:21.467597 546 pending_rounds.cc:76] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 1
I20260806 03:43:21.493337 546 pending_rounds.cc:97] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:43:21.493639 546 raft_consensus.cc:2866] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:21.461836 665 rpcz_store.cc:276] 0806 03:43:19.090850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.091261 (+ 411us) service_pool.cc:227] Handling call
0806 03:43:19.155868 (+ 64607us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:19.808787 (+652919us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.205025 (+1396238us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54000659d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.205412 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1470657280}
I20260806 03:43:21.461568 563 rpcz_store.cc:276] 0806 03:43:20.019402 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.019570 (+ 168us) service_pool.cc:227] Handling call
0806 03:43:20.033086 (+ 13516us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:20.478725 (+445639us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.200163 (+721438us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54000cf654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.200517 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1023322368}
W20260806 03:43:21.462124 669 rpcz_store.cc:269] 0806 03:43:19.514061 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.514206 (+ 145us) service_pool.cc:227] Handling call
0806 03:43:21.214956 (+1700750us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54002c0e54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:21.215234 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":987667200}
W20260806 03:43:21.492420 769 rpcz_store.cc:269] 0806 03:43:20.018321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.018520 (+ 199us) service_pool.cc:227] Handling call
0806 03:43:21.234951 (+1216431us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:21.478779 (+243828us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5430912}
I20260806 03:43:21.463370 770 rpcz_store.cc:276] 0806 03:43:20.005734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.006196 (+ 462us) service_pool.cc:227] Handling call
0806 03:43:21.263434 (+1257238us) spinlock_profiling.cc:243] Waited 79.1 ms on lock 0x7b54002bd9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:21.297715 (+ 34281us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":183828992}
I20260806 03:43:21.499971 562 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:21.461719 663 rpcz_store.cc:276] 0806 03:43:19.977138 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.977298 (+ 160us) service_pool.cc:227] Handling call
0806 03:43:19.977728 (+ 430us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:19.978169 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:19.978206 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:20.000785 (+ 22579us) log.cc:842] Serialized 26 byte log entry
0806 03:43:20.047732 (+ 46947us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:20.047769 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:20.047837 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:21.048563 (+1000726us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:21.203247 (+154684us) raft_consensus.cc:1671] finished
0806 03:43:21.203288 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:21.203806 (+ 518us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4251,"raft.queue_time_us":4001,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":17775,"spinlock_wait_cycles":12265728,"thread_start_us":36374,"threads_started":1,"wal-append.queue_time_us":69363}
I20260806 03:43:21.461212 548 rpcz_store.cc:276] 0806 03:43:20.021882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.027769 (+ 5887us) service_pool.cc:227] Handling call
0806 03:43:20.043276 (+ 15507us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:20.153692 (+110416us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:20.153732 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:20.176418 (+ 22686us) log.cc:842] Serialized 26 byte log entry
0806 03:43:20.232300 (+ 55882us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:20.239427 (+ 7127us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.197171 (+957744us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b5400086454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.197272 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:21.197389 (+ 117us) raft_consensus.cc:1671] finished
0806 03:43:21.197421 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:21.197854 (+ 433us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":215745920,"thread_start_us":38505,"threads_started":1,"wal-append.queue_time_us":94426,"wal-append.run_cpu_time_us":2423,"wal-append.run_wall_time_us":1088308}
I20260806 03:43:21.533456 3400 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:21.536778 3389 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:21.518285 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 902793us
I20260806 03:43:21.552130 3389 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:21.572077 729 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.921s user 0.000s sys 0.058s
W20260806 03:43:21.572829 729 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.922s user 0.000s sys 0.059s
I20260806 03:43:21.573086 3178 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:21.574549 3313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.004s sys 0.007s
I20260806 03:43:21.572064 3275 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.580999 3275 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.582351 3411 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.583395 770 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:21.583240 3411 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.583802 3275 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:21.584192 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:21.586516 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:21.586797 614 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:21.573978 3178 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:21.580116 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.022s sys 0.028s
W20260806 03:43:21.603826 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.962s user 0.023s sys 0.028s
W20260806 03:43:21.575068 3313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.004s sys 0.008s
I20260806 03:43:21.613565 3301 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.614624 3301 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.615227 3178 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:21.615900 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.804s user 0.010s sys 0.052s
W20260806 03:43:21.616302 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.805s user 0.010s sys 0.052s
W20260806 03:43:21.616652 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 804993us
I20260806 03:43:21.617146 3301 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.629248 3329 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:21.629933 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:21.631083 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 997132us
I20260806 03:43:21.463546 401 rpcz_store.cc:276] 0806 03:43:15.341799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:15.350960 (+ 9161us) service_pool.cc:227] Handling call
0806 03:43:21.308749 (+5957789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:18.962570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:18.963175 (+ 605us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:18.963223 (+ 48us) write_op.cc:435] Acquired schema lock
0806 03:43:18.963240 (+ 17us) tablet.cc:660] Decoding operations
0806 03:43:18.963962 (+ 722us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:18.964041 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:18.964083 (+ 42us) tablet.cc:683] Acquiring locks for 4 operations
0806 03:43:18.964632 (+ 549us) tablet.cc:699] Row locks acquired
0806 03:43:18.964655 (+ 23us) write_op.cc:260] PREPARE: finished
0806 03:43:18.964824 (+ 169us) write_op.cc:270] Start()
0806 03:43:18.964941 (+ 117us) write_op.cc:276] Timestamp: P: 1785987798964781 usec, L: 0
0806 03:43:18.964963 (+ 22us) op_driver.cc:331] REPLICATION: starting
0806 03:43:18.965632 (+ 669us) log.cc:842] Serialized 2005 byte log entry
0806 03:43:19.458090 (+492458us) op_driver.cc:444] REPLICATION: finished
0806 03:43:19.736440 (+278350us) write_op.cc:301] APPLY: starting
0806 03:43:19.736523 (+ 83us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:19.736660 (+ 137us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:19.736792 (+ 132us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:19.737414 (+ 622us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:19.737600 (+ 186us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0806 03:43:19.744795 (+ 7195us) write_op.cc:312] APPLY: finished
0806 03:43:19.783177 (+ 38382us) log.cc:842] Serialized 41 byte log entry
0806 03:43:19.813929 (+ 30752us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:19.824706 (+ 10777us) write_op.cc:454] Released schema lock
0806 03:43:19.830831 (+ 6125us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:19.842608 (+ 11777us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:19.842665 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":157076,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":262941,"apply.run_cpu_time_us":3492,"apply.run_wall_time_us":107940,"num_ops":4,"prepare.queue_time_us":408666,"prepare.run_cpu_time_us":3363,"prepare.run_wall_time_us":3364,"replication_time_us":487322,"thread_start_us":462589,"threads_started":2}]]}
I20260806 03:43:21.655793 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:21.617331 3232 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:21.695196 3195 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.629340 3295 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:21.707252 3361 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.679978 3029 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:21.715865 3388 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.014s sys 0.023s
W20260806 03:43:21.715816 3334 consensus_meta.cc:287] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.552s user 0.012s sys 0.008s
I20260806 03:43:21.723814 3206 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.730580 3206 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.733547 3206 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.734202 3206 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.692111 3178 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.735236 663 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:21.735100 3206 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.735836 3178 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.737612 3206 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:21.696298 3275 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:21.719208 2707 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.976s user 0.004s sys 0.046s
W20260806 03:43:21.724301 730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.031s sys 0.053s
I20260806 03:43:21.729481 3334 consensus_queue.cc:1046] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:43:21.745831 730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.105s user 0.032s sys 0.054s
I20260806 03:43:21.707624 3232 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.743871 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:21.743394 3388 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s user 0.016s sys 0.023s
I20260806 03:43:21.743270 615 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:21.743196 2707 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.154s user 0.004s sys 0.046s
I20260806 03:43:21.724397 3394 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.738298 3178 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:21.739127 3275 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.743934 3411 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:21.774142 728 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.912s user 0.015s sys 0.040s
I20260806 03:43:21.742610 3313 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.777882 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:21.756577 3301 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.708223 3361 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.750957 3195 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.791411 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:21.746253 3295 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:21.742615 3329 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.796343 3411 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:21.796511 3295 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.797168 3411 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:21.798044 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2073) took 1108 ms. Trace:
I20260806 03:43:21.798590 668 rpcz_store.cc:276] 0806 03:43:20.689709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.689821 (+ 112us) service_pool.cc:227] Handling call
0806 03:43:21.797774 (+1107953us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54002af154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:21.797966 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1185167616}
I20260806 03:43:21.801498 3313 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.802830 771 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:21.803172 3295 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:21.804495 3313 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:21.804713 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 175051us
I20260806 03:43:21.805142 564 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:21.805828 614 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:21.797221 558 raft_consensus.cc:2932] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:21.808921 3361 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.809572 3361 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.810708 3178 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:21.770438 3365 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:21.792673 728 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.134s user 0.015s sys 0.043s
I20260806 03:43:21.807740 614 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:21.779115 3265 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.034s sys 0.055s
I20260806 03:43:21.745540 3029 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:21.791275 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 84082.3us
I20260806 03:43:21.804497 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1152) took 1197 ms. Trace:
I20260806 03:43:21.798125 3329 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:21.837641 771 rpcz_store.cc:276] 0806 03:43:20.607255 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.607454 (+ 199us) service_pool.cc:227] Handling call
0806 03:43:20.607784 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:21.803375 (+1195591us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.804125 (+ 750us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400184554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.804411 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2747016448}
W20260806 03:43:21.840662 3265 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.034s sys 0.057s
I20260806 03:43:21.845293 3105 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.853062 3275 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.853256 3029 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.794237 3394 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.835096 3411 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.837502 3232 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.863096 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:21.891585 3195 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:21.891839 1668 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:21.862666 501 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:21.869823 3334 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:21.893383 3334 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:21.893458 3394 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:21.893853 3334 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:21.894068 773 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
W20260806 03:43:21.863945 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 56405.6us
I20260806 03:43:21.850603 3332 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:21.896062 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:21.896600 3400 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:21.897229 3400 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.897673 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:21.898731 3232 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:21.862336 3350 consensus_meta.cc:287] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.767s user 0.004s sys 0.014s
W20260806 03:43:21.863402 616 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.67:43937 timed out after 1.698s (SENT) [suppressed 11 similar messages]
I20260806 03:43:21.875250 3178 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.883600 3105 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.845288 3301 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.810324 3361 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.846724 3313 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.870529 3149 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:21.885353 2707 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:21.887221 2872 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:21.888198 3397 consensus_meta.cc:287] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.799s user 0.013s sys 0.009s
I20260806 03:43:21.888454 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 215) took 1913 ms. Trace:
I20260806 03:43:21.898792 3332 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:43:21.872555 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 1.20944e+06us
I20260806 03:43:21.900826 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1056) took 1284 ms. Trace:
I20260806 03:43:21.884039 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:21.912858 3236 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.902191 3329 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.845834 3388 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.921154 3195 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:21.921342 3416 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:21.871829 3365 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:21.923120 3223 consensus_meta.cc:287] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.845s user 0.006s sys 0.019s
W20260806 03:43:21.927235 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 95502.6us
I20260806 03:43:21.928313 2707 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.931349 664 rpcz_store.cc:276] 0806 03:43:19.975253 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:19.976716 (+ 1463us) service_pool.cc:227] Handling call
0806 03:43:20.001171 (+ 24455us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:21.470979 (+1469808us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.888179 (+417200us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540016dad4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.888391 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2773813120}
I20260806 03:43:21.932628 562 rpcz_store.cc:276] 0806 03:43:20.616667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:21.451123 (+834456us) service_pool.cc:227] Handling call
0806 03:43:21.451722 (+ 599us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:21.513830 (+ 62108us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.900588 (+386758us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b540031f6d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:21.900759 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91646720}
I20260806 03:43:21.946943 3416 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.952883 614 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:21.981755 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:21.981982 2707 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:21.982583 3149 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.972285 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1992) took 1351 ms. Trace:
W20260806 03:43:21.981884 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 97963.8us
I20260806 03:43:21.939420 3394 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:21.983481 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:21.978096 3400 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.022982 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:21.945540 3389 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.951030 3178 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:21.984086 775 rpcz_store.cc:276] 0806 03:43:20.620688 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.621199 (+ 511us) service_pool.cc:227] Handling call
0806 03:43:20.621537 (+ 338us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:20.624584 (+ 3047us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:20.624631 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:20.625353 (+ 722us) log.cc:842] Serialized 26 byte log entry
0806 03:43:21.479842 (+854489us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:21.479924 (+ 82us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.480067 (+ 143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:21.947421 (+467354us) raft_consensus.cc:1671] finished
0806 03:43:21.947443 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:21.972186 (+ 24743us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":41710,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":171,"thread_start_us":853735,"threads_started":1,"wal-append.queue_time_us":1273668}
I20260806 03:43:22.029991 3236 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.036089 3422 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.036401 3149 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:22.039086 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 86453.8us
I20260806 03:43:22.040238 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 1985) took 1712 ms. Trace:
I20260806 03:43:22.041016 3416 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.010983 2707 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.042492 3195 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.043300 3394 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:22.045166 3434 log.cc:924] Time spent T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.569s user 0.004s sys 0.001s
I20260806 03:43:22.045850 3334 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.039366 3105 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.048377 3357 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:22.049928 3399 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:22.050458 3223 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:21.947818 3329 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.973002 3313 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.057440 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:21.996320 2872 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:21.999342 3388 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.086632 614 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:22.056123 768 rpcz_store.cc:276] 0806 03:43:20.328112 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.328288 (+ 176us) service_pool.cc:227] Handling call
0806 03:43:20.328764 (+ 476us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:20.329158 (+ 394us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:20.329194 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:20.329674 (+ 480us) log.cc:842] Serialized 26 byte log entry
0806 03:43:20.747678 (+418004us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b4800160158. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:43:20.813041 (+ 65363us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:20.813106 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:20.813205 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:22.039666 (+1226461us) raft_consensus.cc:1671] finished
0806 03:43:22.039707 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:22.039980 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3094,"raft.run_cpu_time_us":10106,"raft.run_wall_time_us":415941,"spinlock_wait_cycles":13260672,"thread_start_us":461743,"threads_started":1,"wal-append.queue_time_us":1691172}
I20260806 03:43:22.076623 3105 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:22.097018 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 295054us
I20260806 03:43:22.091646 3206 raft_consensus.cc:3032] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:22.077678 3397 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.100493 3206 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:22.107026 3361 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.107837 3236 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.107715 3206 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.108786 3389 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.109411 3206 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.109963 3178 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.111153 3195 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.112306 2872 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.109310 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.113763 3195 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.114471 614 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:22.123416 3328 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:22.124253 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.125411 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.125941 614 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:22.126143 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:22.126786 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 73653.2us
I20260806 03:43:22.126972 3332 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:22.127591 3332 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.127950 1662 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:22.128136 3332 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:22.129521 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.129904 3185 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:22.130419 546 raft_consensus.cc:2379] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:22.130540 3185 raft_consensus.cc:686] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:22.131844 3185 consensus_queue.cc:237] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.132227 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.084434 3365 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.139515 3365 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.141950 3365 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.143934 3313 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:22.146422 3174 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.027s user 0.011s sys 0.011s
I20260806 03:43:22.146716 3397 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.147899 3403 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.148542 3403 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.149205 3397 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.118602 3329 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.119432 3105 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.118645 3394 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.120519 548 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:22.120676 3223 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:22.160161 3105 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.120169 3388 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.086542 3422 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.089768 3399 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.079370 2707 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.117245 3334 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:22.103421 1622 connection.cc:232] RPC call timeout handler was delayed by -1.4670873680152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
W20260806 03:43:22.173218 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 395129us
I20260806 03:43:22.163106 3301 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.160755 1657 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:22.165429 3313 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:22.185010 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.185876 1668 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:22.187064 499 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:22.196707 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 442438us
I20260806 03:43:22.203811 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.207088 3419 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.207828 3419 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.208556 3301 consensus_queue.cc:1046] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:22.209762 3301 raft_consensus.cc:3032] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:22.210127 3301 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:22.203420 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.213158 3301 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.213339 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.180941 3178 consensus_queue.cc:1046] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 03:43:22.214198 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.214713 665 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:22.217695 3234 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.241393 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.210167 3419 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.253976 1664 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:22.220474 3206 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:22.260345 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 48790.3us
I20260806 03:43:22.260562 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.253379 498 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:22.222767 3301 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:22.195353 3399 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.265101 769 raft_consensus.cc:2379] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:22.218411 3411 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.272917 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.273514 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.276796 775 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:22.282130 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 65332.1us
I20260806 03:43:22.283385 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:22.285966 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.218047 3313 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:22.290989 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.292034 3313 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.222733 3234 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.310415 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.310070 668 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:22.268950 3403 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.309789 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.316109 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.318953 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:22.321619 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:22.322569 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.322675 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.322916 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:22.323498 1657 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:22.321694 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.318809 772 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:22.325459 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.325953 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:22.327425 501 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:22.326439 773 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:22.289186 1660 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:22.310037 3401 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:43:22.328298 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 110082us
I20260806 03:43:22.340840 501 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:22.343134 3187 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.343672 2911 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:22.344475 3453 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.345095 3453 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.345599 772 raft_consensus.cc:1226] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:22.346172 3178 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:22.346601 498 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:22.347172 3348 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:22.347661 2707 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.342051 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.350483 3399 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.351679 3419 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.351959 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:22.352501 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.353374 669 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:22.353488 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.353890 614 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:22.355099 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.355450 663 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:22.352246 3419 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.357116 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.293907 3206 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.348011 3348 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:22.344496 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.360518 773 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:22.359674 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:22.307638 3411 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.366505 3425 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.339370 3401 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:22.366319 3232 consensus_queue.cc:1046] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:22.343182 2936 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:22.356868 3361 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:22.372007 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 124054us
I20260806 03:43:22.361639 3365 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:43:22.423450 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45875.8us
I20260806 03:43:22.425102 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.356684 3334 consensus_queue.cc:1046] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260806 03:43:22.360042 3445 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.364876 3422 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.372776 3350 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:22.432821 769 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:22.370663 2936 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:22.427759 3265 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.454228 2936 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.454696 3265 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:22.422696 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 51764.4us
I20260806 03:43:22.456686 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:22.457170 3265 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.457684 615 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:22.459144 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.460107 659 raft_consensus.cc:2379] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:22.370733 3401 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:22.373484 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.430219 3232 consensus_queue.cc:1225] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer ada5ef6656cb4420baf77a4b4342909e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:43:22.437093 665 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:22.437592 3328 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:22.439759 3234 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.441102 663 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:22.430639 3403 consensus_queue.cc:1046] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260806 03:43:22.433632 3178 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:22.445526 3206 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.370198 3348 raft_consensus.cc:686] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:22.427197 3365 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:22.360718 2911 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:22.422731 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 57509.6us
I20260806 03:43:22.492041 3388 consensus_queue.cc:1046] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:22.492746 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.494104 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.493829 3348 consensus_queue.cc:237] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.461079 3445 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:22.494894 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 141094us
I20260806 03:43:22.495303 768 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:22.495441 3403 raft_consensus.cc:3032] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:22.495854 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.489388 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.423863 3332 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.508533 667 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:22.554296 3174 raft_consensus.cc:3032] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:22.554970 3174 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:22.556352 3174 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.480201 3232 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:22.422076 3425 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.558627 3232 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:22.559180 3232 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:22.560995 3425 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.562062 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.499539 2911 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:22.509359 3301 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:43:22.549901 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 58837.7us
I20260806 03:43:22.507401 3365 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:22.558175 3174 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.524269 3403 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:22.579952 1668 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:22.569005 2911 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.565505 3450 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.582870 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.572326 3301 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:22.567335 1660 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:22.586346 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:22.588954 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.586300 768 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:22.626935 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.630707 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.645599 664 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:22.589636 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 132003us
I20260806 03:43:22.626151 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.645599 667 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
W20260806 03:43:22.637151 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46865.2us
I20260806 03:43:22.617120 659 raft_consensus.cc:1261] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 03:43:22.651628 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 81964.3us
W20260806 03:43:22.626929 3458 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260806 03:43:22.656049 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.658151 665 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:22.637795 3406 log.cc:924] Time spent T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 03:43:22.616462 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.672642 615 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:22.615937 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.627034 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.589898 3301 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:22.677138 499 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:22.616225 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.693500 3403 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.694566 776 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:22.695313 3403 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.630479 3405 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.120s
I20260806 03:43:22.684033 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.684165 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.637012 3450 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.638034 3411 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.685205 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.624835 3394 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:22.689127 667 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:22.698041 659 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:22.638302 3419 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.593562 3442 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.687037 671 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:22.701947 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.674666 2872 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:22.713038 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:22.714571 664 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:22.717988 664 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:22.724411 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:22.727082 3234 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:22.727514 3234 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:22.727869 3234 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:22.731381 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:22.732609 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 207498us
I20260806 03:43:22.737154 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.743575 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.743739 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.725076 3334 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.684304 3425 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260806 03:43:22.715678 3444 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:22.746054 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.746908 665 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:22.748859 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.708868 3475 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:22.712601 3357 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:22.710409 3361 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.731845 664 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.754508 1622 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:22.754988 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.745079 3334 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.783841 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.710675 3394 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.780678 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.720126 3442 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.785933 664 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:22.810781 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.787026 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.788704 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:22.790800 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.791594 2872 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:22.783820 3419 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.793680 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.832381 671 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:22.832106 3419 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.793968 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.833192 667 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:22.833352 562 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:22.797344 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:22.785295 3332 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.798162 3401 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.835101 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:22.836028 3401 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.803768 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.837462 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.837734 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.839135 544 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260806 03:43:22.805342 3422 raft_consensus.cc:3032] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:22.839502 544 pending_rounds.cc:76] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 3
I20260806 03:43:22.839797 544 pending_rounds.cc:97] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260806 03:43:22.839726 3422 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:22.840291 544 raft_consensus.cc:2866] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:22.830044 3206 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:22.841379 3206 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:22.806463 3410 log.cc:924] Time spent T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.552s user 0.003s sys 0.000s
I20260806 03:43:22.807210 3432 raft_consensus.cc:3032] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:22.786952 674 raft_consensus.cc:1261] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:22.812058 564 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:22.752745 3444 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:22.816294 3361 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.817435 3195 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.818712 3265 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.809798 3232 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.821229 3223 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.858789 3232 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.825321 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.825444 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.859658 3450 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.859481 3265 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.825704 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.860466 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.860716 3332 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.826502 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.853423 3446 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:22.861668 3265 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.861809 3446 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:22.827358 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.862285 3446 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:22.862504 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 599) took 2633 ms. Trace:
I20260806 03:43:22.863051 3394 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.863350 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:22.864140 1657 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:22.834278 3419 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.828135 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.824991 3187 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:22.829833 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.830564 3367 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.826547 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.830506 3357 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.842859 664 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 03:43:22.753650 3475 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:22.849344 3432 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:22.811546 3445 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.748494 3399 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20260806 03:43:22.854676 3401 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.916953 3401 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.857937 564 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:22.918498 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.918998 3401 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.858867 3334 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.919713 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.921118 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.925616 3206 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:22.858626 3223 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.927681 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.928082 3223 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.862286 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.928625 1662 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:22.828951 561 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
W20260806 03:43:22.876403 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 87808.4us
W20260806 03:43:22.874670 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 168494us
I20260806 03:43:22.930858 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.828871 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.931510 3328 raft_consensus.cc:3032] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:22.853564 3444 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:22.931936 3328 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:22.933086 3328 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.894255 3234 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.934124 3418 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.934809 3418 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.935933 3234 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.850386 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.937146 3418 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:22.937493 1660 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:22.938997 728 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:22.940594 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:22.900291 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.941699 1664 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:22.901131 3367 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.862929 1671 rpcz_store.cc:276] 0806 03:43:20.228745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:20.228880 (+ 135us) service_pool.cc:227] Handling call
0806 03:43:20.229180 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:20.284862 (+ 55682us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:20.284894 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:20.287622 (+ 2728us) log.cc:842] Serialized 26 byte log entry
0806 03:43:21.481245 (+1193623us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:21.481314 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:21.481414 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:22.805790 (+1324376us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:22.853195 (+ 47405us) raft_consensus.cc:1671] finished
0806 03:43:22.853222 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:22.862415 (+ 9193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65381,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":30803,"spinlock_wait_cycles":9787776,"thread_start_us":1211885,"threads_started":2,"wal-append.queue_time_us":1866368}
I20260806 03:43:22.890858 664 pending_rounds.cc:76] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Aborting all ops after (but not including) 2
I20260806 03:43:22.867216 3195 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:22.914288 3232 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.976902 546 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:22.915530 3485 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:22.916412 3453 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:22.927770 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.986330 1658 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:22.985970 3485 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.943257 664 pending_rounds.cc:97] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 03:43:22.987450 664 raft_consensus.cc:2866] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:22.922333 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.889328 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.989157 563 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
W20260806 03:43:22.989846 3275 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 03:43:22.990135 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:22.990875 563 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:22.861248 3332 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:22.944770 562 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:22.860747 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:22.983421 548 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:22.989176 564 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:22.957789 499 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:23.001607 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 202377us
I20260806 03:43:22.903164 3475 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.011708 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.012481 563 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:23.013731 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:23.016155 3178 consensus_meta.cc:287] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.518s user 0.010s sys 0.007s
I20260806 03:43:23.016855 3178 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.018841 669 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:43:23.011744 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 54086.6us
I20260806 03:43:22.979206 3450 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:22.990538 3232 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.019804 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.023591 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.036849 499 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.037187 3334 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:23.037628 3334 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.038038 3334 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:23.038795 3361 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.039399 3361 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:23.037652 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.039734 3361 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:23.040096 564 raft_consensus.cc:2379] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:23.040127 3361 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:23.019245 3234 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.042598 3234 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.042987 3234 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.037111 3232 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.023675 3357 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.030686 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.030416 3432 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.032470 1657 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:23.046682 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.047719 3453 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.048117 3453 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:23.048461 3453 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.050930 3485 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.052670 3332 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.052359 2911 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.052845 3490 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.053980 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:23.055038 2936 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.003s
I20260806 03:43:23.055404 3388 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.055538 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:22.994989 3223 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.055907 3388 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.056371 3388 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.056296 3223 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.056701 3490 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:23.058166 3490 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.059749 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.060683 674 raft_consensus.cc:2396] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:23.062656 561 raft_consensus.cc:2354] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:23.033062 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.024516 3397 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260806 03:43:23.064234 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.065132 542 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:23.065337 3453 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.033946 3195 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.030455 3367 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.035202 3328 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.055429 2936 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.067525 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.032672 3423 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:23.023857 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.070628 499 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:23.071197 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 57359.5us
I20260806 03:43:23.075899 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.091807 665 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.075994 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:23.076439 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.070192 3442 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.077061 2936 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.077767 547 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.078156 674 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.077069 1663 raft_consensus.cc:1226] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.081471 3234 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.116933 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:23.099862 3367 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:23.118844 3234 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.118949 3367 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.119403 3234 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.119854 3234 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.119984 3367 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.120951 3357 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:23.121924 3234 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.082823 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.123245 3453 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.123814 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.124119 3453 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.124758 3453 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.026593 3450 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:23.105762 3365 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.517s user 0.019s sys 0.006s
I20260806 03:43:23.125874 3450 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:23.126215 3365 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.121730 3357 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.123294 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.130671 663 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:23.100194 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.108858 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.131083 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.131660 659 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:23.068730 3223 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.113931 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.132651 665 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:23.111639 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.099997 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.095785 3432 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:23.099916 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.131738 671 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:23.073010 2911 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.124696 561 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:23.066177 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:23.139412 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 115142us
I20260806 03:43:23.092346 3490 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:23.139284 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 456319us
I20260806 03:43:23.143680 616 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:23.103626 3174 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.521s user 0.007s sys 0.003s
I20260806 03:43:23.144901 615 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.151659 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.152383 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.151566 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.152501 664 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.153575 659 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.144004 1658 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:23.155397 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.155464 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.157755 665 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:23.143682 3485 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.170279 498 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:23.166145 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.167999 3332 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.174855 3361 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.177148 3361 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.141256 3365 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.168566 615 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.145658 3403 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.140188 3453 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.194872 615 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.186407 3365 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.149425 3490 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.155896 542 raft_consensus.cc:2362] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2. Re-sending same reply.
I20260806 03:43:23.178812 3174 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.141021 3444 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.192922 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.202585 3357 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.204020 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.149951 3232 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260806 03:43:23.204533 3388 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.205469 3334 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.169739 2936 consensus_queue.cc:1046] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 03:43:23.184952 3403 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.207414 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:23.208719 498 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.209650 3234 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:23.216298 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:23.217479 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 162023us
I20260806 03:43:23.245090 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.225062 3174 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.249895 558 raft_consensus.cc:2379] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:23.246939 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.227913 3178 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.226742 614 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.179171 3485 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:23.231680 671 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:23.232270 616 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:23.221215 3301 consensus_meta.cc:287] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.505s user 0.004s sys 0.015s
I20260806 03:43:23.233073 664 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.238087 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.238768 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.211630 3365 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.242888 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.252211 3419 raft_consensus.cc:3032] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:23.246124 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.193838 3450 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:23.247948 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 69035.8us
I20260806 03:43:23.237761 3223 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.237941 1657 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.246984 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.226964 659 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:23.254387 3357 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:23.254478 3399 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.254716 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.259661 3490 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.258970 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.267220 3178 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.288846 3178 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.289279 3178 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.252434 3174 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.294319 776 raft_consensus.cc:2354] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6.
I20260806 03:43:23.252151 3504 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:23.260376 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 51731us
I20260806 03:43:23.261027 3444 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.246042 3403 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.261575 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.277452 3399 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.299908 3399 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.243288 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.305752 775 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.307801 3334 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.310117 3422 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.310739 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.269589 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.272258 767 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:23.280148 615 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.255026 3485 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.282871 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.264252 3411 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:23.259567 1665 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:23.286868 561 raft_consensus.cc:1261] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:23.269079 3365 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.287045 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.265832 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.268697 3419 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.246778 3234 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.250908 3475 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.262254 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.381294 775 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:23.297863 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:23.383620 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 85880.3us
I20260806 03:43:23.384680 727 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.317339 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.268249 3301 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.403194 661 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:23.404366 3301 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.345047 3450 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:23.335188 400 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35370 (request call id 318) took 11728 ms (client timeout 15000 ms). Trace:
I20260806 03:43:23.295598 3504 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.344661 3223 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.369477 768 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:23.344481 498 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.371707 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:23.374888 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 92082.5us
I20260806 03:43:23.385820 771 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 3
W20260806 03:43:23.386665 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 116686us
I20260806 03:43:23.389586 1657 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:23.389973 3403 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.392901 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.378006 3234 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:23.382543 3185 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.395624 3453 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:23.397019 3178 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.389219 3419 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.377784 1662 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.381883 1671 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:23.409561 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.292421 542 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:23.413668 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.425654 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.424940 727 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.426771 616 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.427706 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.427874 616 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.428776 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.413833 615 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.414489 667 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.416221 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.333782 3446 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.412489 3450 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:23.420403 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 75977.4us
I20260806 03:43:23.424695 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.412308 3504 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.419023 3334 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:43:23.430850 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 213941us
I20260806 03:43:23.432804 547 raft_consensus.cc:1261] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:23.433303 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:23.432765 768 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.432281 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.436807 614 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.343379 3444 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260806 03:43:23.397061 3174 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.447566 3442 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:23.430742 3453 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.448290 768 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:23.381510 3499 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260806 03:43:23.448455 3403 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.440991 3332 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:23.438160 3411 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.435557 3275 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.432657 3490 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.438530 3328 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.449045 3301 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.449880 3388 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.450053 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.511943 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:23.405459 400 rpcz_store.cc:269] 0806 03:43:11.588647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:11.588868 (+ 221us) service_pool.cc:227] Handling call
0806 03:43:23.316711 (+11727843us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:21.862190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:21.862963 (+ 773us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:21.863025 (+ 62us) write_op.cc:435] Acquired schema lock
0806 03:43:21.863048 (+ 23us) tablet.cc:660] Decoding operations
0806 03:43:21.863595 (+ 547us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:21.863716 (+ 121us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:21.863770 (+ 54us) tablet.cc:683] Acquiring locks for 2 operations
0806 03:43:21.864234 (+ 464us) tablet.cc:699] Row locks acquired
0806 03:43:21.864269 (+ 35us) write_op.cc:260] PREPARE: finished
0806 03:43:21.864434 (+ 165us) write_op.cc:270] Start()
0806 03:43:21.864620 (+ 186us) write_op.cc:276] Timestamp: P: 1785987801864365 usec, L: 0
0806 03:43:21.864662 (+ 42us) op_driver.cc:331] REPLICATION: starting
0806 03:43:21.865606 (+ 944us) log.cc:842] Serialized 929 byte log entry
0806 03:43:22.159402 (+293796us) spinlock_profiling.cc:243] Waited 5.13 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f32133d95fe 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb
0806 03:43:22.617425 (+458023us) op_driver.cc:444] REPLICATION: finished
0806 03:43:22.740202 (+122777us) write_op.cc:301] APPLY: starting
0806 03:43:22.740287 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:22.740392 (+ 105us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:22.740420 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:22.741138 (+ 718us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:22.741529 (+ 391us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 03:43:22.741588 (+ 59us) write_op.cc:312] APPLY: finished
0806 03:43:22.741958 (+ 370us) log.cc:842] Serialized 29 byte log entry
0806 03:43:22.742173 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:22.742298 (+ 125us) write_op.cc:454] Released schema lock
0806 03:43:22.742393 (+ 95us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:22.742744 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:22.742772 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4986,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":122379,"apply.run_cpu_time_us":2859,"apply.run_wall_time_us":2860,"num_ops":2,"prepare.queue_time_us":332888,"prepare.run_cpu_time_us":36634,"prepare.run_wall_time_us":470464,"raft.queue_time_us":132938,"raft.run_cpu_time_us":38185,"raft.run_wall_time_us":134568,"replication_time_us":752643,"spinlock_wait_cycles":11836544,"thread_start_us":799227,"threads_started":4,"wal-append.queue_time_us":496181}]]}
I20260806 03:43:23.441882 499 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.454836 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.467118 3450 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.437790 3185 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:23.455601 3509 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260806 03:43:23.536600 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 94978.4us
I20260806 03:43:23.343580 3507 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
I20260806 03:43:23.466099 771 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.476430 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.483180 3448 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:23.483716 773 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:23.483976 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.484577 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.485185 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.487596 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.538820 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.539716 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:23.540021 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 52504.6us
I20260806 03:43:23.540750 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.418660 3425 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260806 03:43:23.449307 3357 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:23.431412 3223 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:23.492193 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 55460.2us
I20260806 03:43:23.512847 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.487454 3485 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.477895 3453 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:23.518096 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 76294.6us
I20260806 03:43:23.520107 3475 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.523574 3422 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:43:23.524039 3513 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260806 03:43:23.518796 3301 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.592307 3301 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:23.525272 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 70520.2us
I20260806 03:43:23.525813 3367 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:23.594223 3485 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.594626 3367 raft_consensus.cc:686] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.595119 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.595525 3367 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.596323 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.556314 3265 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.597463 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.526576 3361 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.489873 3419 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.466917 3442 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.526629 3526 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.601521 3526 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.545979 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.603593 3526 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.604470 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:23.551131 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:23.552886 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.509841 3328 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.514144 3490 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.556382 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.557806 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.577886 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.559376 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.585901 664 raft_consensus.cc:2379] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:23.584307 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.613090 771 raft_consensus.cc:2379] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:23.585192 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:23.596521 3510 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260806 03:43:23.589248 3453 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.600565 3442 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.456766 3446 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.544744 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1620) took 1086 ms. Trace:
I20260806 03:43:23.557168 401 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:23.609970 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 71245.5us
I20260806 03:43:23.611018 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.594247 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:23.614295 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56571.8us
I20260806 03:43:23.613695 768 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:23.584275 2936 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.617802 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.621728 669 rpcz_store.cc:276] 0806 03:43:22.458842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:22.469490 (+ 10648us) service_pool.cc:227] Handling call
0806 03:43:22.489604 (+ 20114us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:23.019211 (+529607us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:23.544489 (+525278us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400387054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:23.544668 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1216594816}
I20260806 03:43:23.581720 3357 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.617655 773 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:23.619311 616 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.654754 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.655314 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.543562 3399 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.652789 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:23.615434 3328 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.654023 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.654273 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:23.655709 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 61530.1us
I20260806 03:43:23.621836 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.658812 501 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.658841 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.659531 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.661281 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.616842 3334 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.663946 773 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.619247 3475 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:23.664625 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.662667 664 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:23.665072 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.668656 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.615370 3490 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.672089 615 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.657855 3357 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:23.657138 3499 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.675242 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.672437 401 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 4 to 5, VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) evicted. New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:23.673951 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.688321 669 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.660115 3178 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.693519 498 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.676542 775 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:23.666667 3265 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:23.666812 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.662096 3485 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:23.662315 2936 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:23.695560 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.702395 664 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:23.697431 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.703547 665 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:23.665587 2707 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:23.669171 3223 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.670977 3334 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.701082 3454 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:23.676290 2911 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.710170 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.714141 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.718452 564 raft_consensus.cc:1261] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:43:23.691712 3174 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.720706 1622 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.723989 664 raft_consensus.cc:1226] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:23.725033 665 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.673846 3444 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 03:43:23.671869 3361 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260806 03:43:23.707486 3187 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:23.716114 401 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 4 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:23.734628 401 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 3, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 3 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:23.710724 3223 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.712137 3334 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.717000 3265 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:23.718679 3507 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.718740 2911 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.707355 2707 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.748492 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.752068 667 raft_consensus.cc:2379] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:23.753160 667 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:23.723320 3432 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:23.732925 3524 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:23.732092 3454 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.735976 401 catalog_manager.cc:5495] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: config changed from index 3 to 4, 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER ada5ef6656cb4420baf77a4b4342909e (127.30.138.65), VOTER d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66), VOTER 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67), VOTER 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:23.595134 3525 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:23.758219 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 64727.9us
I20260806 03:43:23.760471 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1629) took 1057 ms. Trace:
I20260806 03:43:23.761801 663 rpcz_store.cc:276] 0806 03:43:22.703395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:22.703529 (+ 134us) service_pool.cc:227] Handling call
0806 03:43:22.716247 (+ 12718us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:23.131068 (+414821us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:23.760252 (+629184us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b540019afd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:23.760412 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":952612736}
I20260806 03:43:23.762442 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:23.738409 3223 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.742748 3265 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.719077 3526 consensus_queue.cc:1046] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 03:43:23.698293 3523 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:23.771590 667 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:23.755924 2911 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.742575 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.769333 498 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:23.755610 2707 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:23.766201 401 catalog_manager.cc:5495] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 4 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:23.788316 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2093) took 1104 ms. Trace:
I20260806 03:43:23.790693 498 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.764384 3454 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.793149 667 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:23.712327 3520 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.249s
I20260806 03:43:23.788735 661 rpcz_store.cc:276] 0806 03:43:22.684334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:22.684979 (+ 645us) service_pool.cc:227] Handling call
0806 03:43:22.721050 (+ 36071us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:23.403645 (+682595us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:23.788067 (+384422us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b54000b2054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:23.788254 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568279296}
I20260806 03:43:23.770342 3432 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:23.793661 401 catalog_manager.cc:5495] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 5, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:23.794160 498 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:23.758523 3397 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 03:43:23.772957 3265 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.782306 1662 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:23.767248 3394 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 03:43:23.803256 401 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 3, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 3 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:23.795117 3454 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.807806 562 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.826066 1671 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.826152 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.827036 562 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:23.827857 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.828883 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.829061 3450 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:23.809484 3417 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:23.810364 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.811017 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.830747 3524 consensus_queue.cc:1046] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:23.831066 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.831869 3524 raft_consensus.cc:3032] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:23.832211 3524 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:23.793812 3361 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:23.833864 3361 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.834246 3361 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.835801 3507 raft_consensus.cc:3032] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:23.836201 3507 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:23.836644 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.837065 3507 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.796674 3444 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:23.812880 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.813611 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.814539 3195 consensus_queue.cc:1046] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:23.815326 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.816123 3275 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:23.860260 564 raft_consensus.cc:1226] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:23.757616 665 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:23.861310 665 raft_consensus.cc:595] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:23.861660 665 raft_consensus.cc:603] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:23.818816 3394 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:23.805083 3446 raft_consensus.cc:2741] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:43:23.793365 3445 consensus_queue.cc:1046] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260806 03:43:23.823616 3403 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:23.864033 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.811764 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.833056 3524 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.812304 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:23.807904 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.849454 776 raft_consensus.cc:1261] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:23.843557 669 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:23.798092 3432 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:23.838479 401 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:23.909992 401 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:23.912017 401 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 2, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 2 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:23.841226 499 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:23.803807 3533 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:23.821365 3301 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.865572 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.806038 3399 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:23.866977 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.858052 2936 raft_consensus.cc:686] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.930063 659 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:23.838352 3507 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.937280 766 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:23.939720 3515 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:23.855604 3418 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.941360 3418 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.915383 3432 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:23.943614 3418 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.944878 3518 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.945431 3518 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.949294 3527 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:23.952337 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.953322 3223 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.954568 3533 raft_consensus.cc:686] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.956037 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:23.955756 3533 consensus_queue.cc:237] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.789769 3523 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.904990 3361 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.959860 3444 raft_consensus.cc:3032] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:23.960403 3444 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:23.960526 3523 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:23.961297 3361 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.962597 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.963111 3429 raft_consensus.cc:3032] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:23.963644 3429 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:23.864894 3275 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.881188 3540 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.964543 3540 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.964634 3429 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.965476 3185 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.965947 3429 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:23.891556 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.894604 548 raft_consensus.cc:2379] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:23.894735 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:23.967674 665 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.967927 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:23.968492 548 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:43:23.895457 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 03:43:23.968529 768 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:23.859925 3504 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.859690 3334 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.961383 3444 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.855953 3365 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:23.897763 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.898144 547 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:23.990361 671 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.900931 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:23.865125 3516 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.150s user 0.001s sys 0.000s
I20260806 03:43:23.871248 3403 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:23.902060 1668 raft_consensus.cc:2354] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:23.902933 3524 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:23.856421 2911 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.907972 664 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:23.842334 773 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:23.863310 3394 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:23.863271 3174 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:23.927237 770 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:23.928719 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:23.921669 3520 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.923031 561 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:23.931241 1622 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:23.933899 3301 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.934767 2936 consensus_queue.cc:237] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:23.954545 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 113444us
I20260806 03:43:23.971680 1624 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:24.019150 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 88033.1us
W20260806 03:43:24.019439 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 47844.3us
I20260806 03:43:24.020141 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.972069 776 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:23.870065 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.021569 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.022799 661 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:23.978750 669 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.971210 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:43:23.979553 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 305662us
I20260806 03:43:23.896667 3234 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.882254 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:23.965996 3185 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:23.995702 769 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:23.967685 3195 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:24.001029 616 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:24.001662 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 276106us
W20260806 03:43:24.045089 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 44187.6us
I20260806 03:43:24.046038 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.047001 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:23.967229 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.015285 546 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:24.015409 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:23.987222 3518 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:23.996178 3527 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:23.916023 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:24.029214 563 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:24.033147 674 consensus_meta.cc:287] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.000s
I20260806 03:43:24.035190 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:24.040518 664 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:43:24.043185 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:24.052448 1668 raft_consensus.cc:1226] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:24.053117 615 leader_election.cc:398] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:24.058527 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.066879 615 leader_election.cc:401] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:43:24.057789 3332 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:24.059414 3419 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260806 03:43:24.064160 558 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:24.030066 3397 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:23.996655 3453 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.029572 548 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:24.031669 3444 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:24.096148 3504 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:24.096750 3265 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.097771 3419 consensus_queue.cc:1046] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:24.099016 3419 raft_consensus.cc:3032] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:24.099382 3419 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:24.096058 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 49136.5us
I20260806 03:43:24.100641 727 leader_election.cc:398] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:24.100229 3419 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.100948 727 leader_election.cc:401] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:24.101500 3419 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:24.103864 665 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:24.107034 727 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.067159 547 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:24.045233 3446 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.070822 1622 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:24.074851 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 52551.6us
I20260806 03:43:24.069424 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:24.065493 674 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:24.033676 3540 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.014873 3174 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.058668 3301 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:24.059723 2872 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:24.079892 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:24.082744 3185 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.083853 3275 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:24.085158 3234 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.086242 3543 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.062628 3526 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:24.066272 3454 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.068579 766 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:24.063107 3403 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.131568 3301 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:24.092612 3490 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.093032 3223 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.089035 3432 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.058241 3520 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.094192 3417 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:24.078465 3394 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.075399 3527 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.095685 3453 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:24.112636 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.119246 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.187230 771 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:24.187588 770 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.080181 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.123024 661 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:24.064517 3195 raft_consensus.cc:686] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:24.068035 3332 raft_consensus.cc:686] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:24.121132 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 50418.3us
I20260806 03:43:24.128989 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.193328 776 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 1.
I20260806 03:43:24.077013 3445 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:24.081985 3361 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 03:43:24.130278 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.120138 3178 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:24.147459 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:24.120406 3504 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:24.087566 3489 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:24.123658 3515 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:24.170651 562 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:24.170138 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1082) took 1001 ms. Trace:
W20260806 03:43:24.172811 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 106131us
I20260806 03:43:24.105702 3357 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.128198 3328 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.158877 3507 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:24.194525 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.196652 564 raft_consensus.cc:2354] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:24.197674 615 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:24.179492 3301 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.183717 3265 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.205385 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.205999 558 rpcz_store.cc:276] 0806 03:43:23.168751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:23.250734 (+ 81983us) service_pool.cc:227] Handling call
0806 03:43:24.169957 (+919223us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540011e9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:24.170081 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1721547648}
I20260806 03:43:24.186575 3403 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:24.264212 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 58951.9us
I20260806 03:43:24.264991 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.266163 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.208335 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.190563 3334 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.203842 2911 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:24.227489 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 97317.7us
I20260806 03:43:24.215320 3394 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:24.196820 3365 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:24.219707 3223 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:24.210081 1665 raft_consensus.cc:1226] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:24.256466 767 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:24.256742 561 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 03:43:24.257001 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.291280 661 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:24.257108 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.203369 3489 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:24.284808 3547 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260806 03:43:24.255992 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.323817 674 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.212050 3540 raft_consensus.cc:3032] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:24.247227 3195 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.271287 770 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:24.205410 3523 raft_consensus.cc:2741] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260806 03:43:24.256093 3518 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.255975 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:24.275331 3515 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260806 03:43:24.277630 3553 consensus_queue.cc:1046] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260806 03:43:24.295141 2872 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:24.283397 3397 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.296324 3357 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.303108 3527 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.304430 3328 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:24.305039 3490 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:24.305969 3454 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:24.268100 3448 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 03:43:24.284581 3546 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260806 03:43:24.310148 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.231356 3332 consensus_queue.cc:237] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.274963 3446 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:24.330832 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.290539 3223 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:24.365540 3223 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:24.369427 3549 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.115s user 0.002s sys 0.000s
I20260806 03:43:24.350615 3523 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260806 03:43:24.370886 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:24.371191 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.371804 668 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:24.353533 3450 consensus_queue.cc:1046] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260806 03:43:24.366174 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.372718 3450 raft_consensus.cc:3032] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:24.373138 3450 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:24.373215 776 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:24.375705 659 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:24.377472 659 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:24.377941 616 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.335522 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.293118 664 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:24.340482 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 297406us
I20260806 03:43:24.342608 663 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:43:24.322427 3489 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:24.348376 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 82286.9us
I20260806 03:43:24.326007 3540 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:24.283639 2707 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.349687 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:24.341612 3504 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260806 03:43:24.356834 771 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:24.342902 3174 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.364801 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.365027 3485 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:24.349287 3265 raft_consensus.cc:3032] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:24.408159 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.409144 770 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
W20260806 03:43:24.410712 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61104.6us
I20260806 03:43:24.411581 727 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.412804 501 leader_election.cc:398] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6.
I20260806 03:43:24.406548 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.389518 674 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:24.413749 668 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:24.409222 3265 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:24.418618 3454 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:24.418192 3265 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.419054 3454 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:24.419435 3454 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:24.419589 3265 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.422142 498 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:24.414714 3548 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.229s user 0.000s sys 0.004s
I20260806 03:43:24.424536 1622 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:24.426234 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:24.359628 3432 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.424520 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:24.405504 3518 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:24.429780 3570 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.432621 667 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:24.432744 770 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:24.451812 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.433399 3394 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:24.404654 3357 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260806 03:43:24.435706 1663 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:24.373996 3450 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.424242 3485 raft_consensus.cc:686] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:24.504875 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:24.421340 3523 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:24.458897 664 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:24.487352 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.445380 727 leader_election.cc:398] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:24.506812 3187 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:24.507134 727 leader_election.cc:401] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260806 03:43:24.508189 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 62943us
I20260806 03:43:24.446169 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:24.509519 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 63437.4us
I20260806 03:43:24.509588 614 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:24.510015 3332 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:24.510432 3328 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:43:24.510651 3178 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:24.511086 3178 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:24.511162 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:24.511534 3178 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.512136 3275 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:24.512516 3275 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:24.512887 3275 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.514567 3185 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:24.437760 2872 raft_consensus.cc:3032] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:24.515266 3399 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:24.516775 3453 raft_consensus.cc:3032] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:24.517220 3453 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:24.515431 2872 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:24.518116 3453 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.507745 3187 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.510448 3332 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:24.519620 3453 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.519776 3332 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:24.521059 3187 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:24.514576 3490 consensus_queue.cc:1046] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260806 03:43:24.524132 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:24.525310 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:24.524143 3523 raft_consensus.cc:3032] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:24.528798 3523 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:24.379144 3520 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:24.531980 498 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:24.467712 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.522755 3275 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.537679 768 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:24.466262 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.544996 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.545439 770 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:24.451292 3518 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:24.545879 3518 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.546083 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.546907 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.547551 558 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:24.546840 769 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:24.408509 3442 consensus_queue.cc:1046] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260806 03:43:24.453675 3507 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:24.567051 674 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:24.457319 3524 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:24.565173 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 54088.8us
I20260806 03:43:24.458166 3553 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:24.574550 771 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:24.562326 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.455886 3432 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:24.453284 3397 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260806 03:43:24.541724 3444 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:24.549373 3187 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:24.532850 3394 raft_consensus.cc:686] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:24.590912 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.548168 3485 consensus_queue.cc:237] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:24.592854 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 67625us
I20260806 03:43:24.456903 3422 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20260806 03:43:24.601545 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.566520 3507 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:24.529307 3332 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.572144 3524 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:24.589933 3450 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:24.621449 3507 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:24.504385 3540 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.604008 3429 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:24.604525 3583 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:24.626180 3524 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:24.704959 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:24.710332 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 136464us
I20260806 03:43:24.653502 2707 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:24.585976 3523 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.756331 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:24.705341 3403 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:24.768725 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.799541 564 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:24.769846 563 raft_consensus.cc:1226] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:43:24.797112 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.803558 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.803994 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.804342 548 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:24.805483 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:24.805720 3574 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260806 03:43:24.806286 548 raft_consensus.cc:2379] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:24.806551 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.807608 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.807797 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.808579 542 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:24.808907 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:24.809583 548 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:24.668377 3419 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260806 03:43:24.767980 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.812629 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.820495 563 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:24.843201 501 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.845788 3444 raft_consensus.cc:686] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:24.692608 3361 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260806 03:43:24.668207 3561 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.125s
I20260806 03:43:24.703156 3429 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:24.880365 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 63514.7us
I20260806 03:43:24.847246 3444 consensus_queue.cc:237] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.869228 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:24.768478 3583 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:24.883270 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 281757us
I20260806 03:43:24.883985 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:24.883891 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 298192us
I20260806 03:43:24.896560 544 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:24.766690 3301 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:24.848840 3187 raft_consensus.cc:686] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:24.905736 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:24.956961 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 51385.2us
I20260806 03:43:24.958729 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:24.952760 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 71319.5us
I20260806 03:43:24.953647 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:24.964932 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:24.880717 3523 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:24.968855 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.754002 3589 consensus_queue.cc:1046] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.182s
I20260806 03:43:24.880726 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 526) took 1085 ms. Trace:
I20260806 03:43:24.990909 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:24.891093 3275 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.018853 769 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:25.019608 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.011590 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.020655 674 raft_consensus.cc:2379] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:25.020587 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.021095 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.012838 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.013957 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.014232 671 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:24.792649 3394 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.951282 3553 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:25.030001 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 375) took 1430 ms. Trace:
I20260806 03:43:24.987741 3504 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:25.024626 661 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:25.030454 767 rpcz_store.cc:276] 0806 03:43:23.599909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:23.617181 (+ 17272us) service_pool.cc:227] Handling call
0806 03:43:23.653548 (+ 36367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:24.290504 (+636956us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:25.029760 (+739256us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b5400320854. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:25.029939 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1318156032}
I20260806 03:43:25.023188 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:24.817145 3403 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:24.890892 3367 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20260806 03:43:24.882457 3399 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.125s
I20260806 03:43:25.021677 544 raft_consensus.cc:2362] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3. Re-sending same reply.
I20260806 03:43:25.022356 665 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:24.980428 2707 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:25.106945 3332 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.152555 769 raft_consensus.cc:1261] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 03:43:25.160884 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 138721us
W20260806 03:43:25.107506 3516 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 03:43:25.031553 3527 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:25.168377 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.107932 766 raft_consensus.cc:1261] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:25.166934 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:25.172403 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:25.107210 2707 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.134716 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.180706 727 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:25.182845 562 raft_consensus.cc:1226] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260806 03:43:25.188208 3576 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.306s user 0.001s sys 0.000s
I20260806 03:43:25.188314 3580 consensus_queue.cc:1046] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.161s
W20260806 03:43:25.189627 3581 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 03:43:25.190604 3596 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:25.191264 3596 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:25.191819 3596 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:25.194626 3586 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:25.197150 3453 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.677s user 0.011s sys 0.000s
W20260806 03:43:25.203258 3223 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.837s user 0.009s sys 0.001s
I20260806 03:43:25.202594 3187 consensus_queue.cc:237] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.203667 3223 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:25.204933 3265 consensus_meta.cc:287] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.775s user 0.009s sys 0.000s
I20260806 03:43:25.205709 546 raft_consensus.cc:2396] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:25.174108 3365 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 03:43:25.159451 3589 raft_consensus.cc:3032] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:25.206637 3589 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:25.207059 3403 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.207782 3589 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.209609 3589 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:25.205729 3223 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.185480 3527 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.211468 665 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:25.213176 3527 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.214154 563 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:25.215329 542 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:43:25.216332 1657 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.760s user 0.011s sys 0.000s
I20260806 03:43:25.216725 1657 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
W20260806 03:43:25.217677 3450 consensus_meta.cc:287] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.564s user 0.002s sys 0.006s
I20260806 03:43:25.219085 564 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:25.219353 3301 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:25.177516 3178 consensus_meta.cc:287] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.666s user 0.007s sys 0.004s
I20260806 03:43:25.226994 3178 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.228192 3584 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260806 03:43:25.226135 558 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:25.009199 1665 rpcz_store.cc:276] 0806 03:43:23.783447 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:23.791003 (+ 7556us) service_pool.cc:227] Handling call
0806 03:43:23.803480 (+ 12477us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:24.308476 (+504996us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:24.798470 (+489994us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54004f5e54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:24.868699 (+ 70229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881116416}
W20260806 03:43:25.285015 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 442032us
I20260806 03:43:25.185989 3599 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:25.189781 3580 raft_consensus.cc:3032] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:43:25.269294 3328 consensus_meta.cc:287] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.758s user 0.009s sys 0.003s
I20260806 03:43:25.251931 3417 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:25.282327 3454 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.862s user 0.010s sys 0.003s
W20260806 03:43:25.300328 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 43858.1us
I20260806 03:43:25.253552 3453 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:25.294884 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 135029us
I20260806 03:43:25.253039 3450 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:25.336894 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:25.303217 501 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.285705 3301 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.301654 3527 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:25.365865 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 63014.3us
I20260806 03:43:25.302731 3580 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:25.371184 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:25.371927 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.315955 3328 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
W20260806 03:43:25.356426 3507 consensus_meta.cc:287] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.688s user 0.009s sys 0.001s
I20260806 03:43:25.375409 665 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:25.318084 3429 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:25.391469 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:25.338408 3610 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:25.394876 768 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.853s user 0.006s sys 0.007s
I20260806 03:43:25.353828 3223 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.355675 3178 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.300120 3591 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260806 03:43:25.360891 3453 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.401091 768 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:25.368551 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.381899 3444 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:25.408785 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 163637us
I20260806 03:43:25.373413 3561 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.253520 3583 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.420866 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:25.342991 3454 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.430871 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.386940 3586 raft_consensus.cc:686] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:25.431847 661 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:25.359382 3450 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.396842 3328 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:25.442242 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 73817.7us
I20260806 03:43:25.438170 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:25.399921 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:25.367245 3301 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.311316 3553 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.384279 3527 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.325625 3417 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.449937 775 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:25.395095 3507 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.453225 772 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:25.443367 501 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.435644 3178 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.395830 3265 consensus_queue.cc:1046] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 03:43:25.458796 768 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
W20260806 03:43:25.445765 3432 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.777s user 0.032s sys 0.051s
I20260806 03:43:25.447153 3444 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:25.470113 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:25.417044 3223 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.512223 3223 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.447335 2707 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.443662 3580 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.454663 3561 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.516844 3561 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.464578 3429 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:25.473634 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:25.520263 3594 consensus_queue.cc:1046] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260806 03:43:25.521546 3594 raft_consensus.cc:3032] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:43:25.475569 3432 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.821s user 0.032s sys 0.052s
I20260806 03:43:25.522557 3432 raft_consensus.cc:686] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:25.523245 3507 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:25.523810 3432 consensus_queue.cc:237] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.481509 3615 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.482911 3527 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.485447 3417 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:25.486402 3611 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.472136 3178 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.525481 3615 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.526000 3611 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.527993 3178 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.528156 3611 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:25.472426 3489 consensus_meta.cc:287] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.052s user 0.012s sys 0.001s
I20260806 03:43:25.488332 3301 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:25.488830 3572 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.940s user 0.013s sys 0.051s
I20260806 03:43:25.492117 3616 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:25.468497 3604 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:25.439028 3453 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:25.497478 3524 consensus_meta.cc:287] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.731s user 0.009s sys 0.000s
W20260806 03:43:25.498814 3605 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.323s user 0.001s sys 0.000s
I20260806 03:43:25.500406 3454 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:25.500728 3540 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:25.501242 1663 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 03:43:25.562541 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 117055us
W20260806 03:43:25.507495 3518 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.961s user 0.011s sys 0.001s
W20260806 03:43:25.509647 671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2123) took 1528 ms (client timeout 2009 ms). Trace:
I20260806 03:43:25.505214 776 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:25.477380 3450 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.479446 3586 consensus_queue.cc:237] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.529129 3489 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:25.529301 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 45180.1us
I20260806 03:43:25.542274 2872 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.521898 3594 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:25.530112 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56573.8us
W20260806 03:43:25.558081 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 54315.9us
I20260806 03:43:25.448580 3328 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.565905 3367 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:25.563727 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 94133.1us
I20260806 03:43:25.566020 3604 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:25.568080 671 rpcz_store.cc:269] 0806 03:43:23.981046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:23.991235 (+ 10189us) service_pool.cc:227] Handling call
0806 03:43:25.509362 (+1518127us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b540036d3d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:25.509561 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2205435776}
I20260806 03:43:25.558596 3527 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:25.494127 3596 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.581920 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.517824 3429 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:25.586660 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.585318 772 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:25.608253 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:25.645907 1664 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:25.646166 768 raft_consensus.cc:2379] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:25.648490 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.650667 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:25.651156 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.651918 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:25.652189 664 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:25.652940 2872 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:25.564550 3524 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.653553 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.655107 3524 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.655791 667 raft_consensus.cc:2354] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4.
I20260806 03:43:25.656272 3596 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.658043 3453 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.646944 3429 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:25.659713 3328 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:25.651360 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.660557 661 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
W20260806 03:43:25.660840 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 74678.5us
W20260806 03:43:25.668581 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.019s sys 0.011s
W20260806 03:43:25.669399 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.019s sys 0.011s
W20260806 03:43:25.671015 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 83234.6us
I20260806 03:43:25.672933 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.571341 3518 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.674077 768 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:25.677081 499 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:25.686182 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:25.687498 768 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:25.690963 616 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:25.693192 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 512537us
I20260806 03:43:25.694200 3604 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.695822 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:25.695715 3450 consensus_queue.cc:1046] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:25.696970 3422 raft_consensus.cc:3032] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:25.697098 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.698077 3453 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:25.645428 3609 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:25.584224 3637 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.698928 3617 consensus_queue.cc:1046] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:25.696983 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2047) took 1213 ms. Trace:
I20260806 03:43:25.699797 3637 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.700739 773 rpcz_store.cc:276] 0806 03:43:24.483180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.483350 (+ 170us) service_pool.cc:227] Handling call
0806 03:43:24.483833 (+ 483us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:24.617437 (+133604us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:24.617478 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:24.617987 (+ 509us) log.cc:842] Serialized 26 byte log entry
0806 03:43:24.872449 (+254462us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:24.872508 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:24.872617 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:25.696681 (+824064us) raft_consensus.cc:1671] finished
0806 03:43:25.696712 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:25.696906 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":131489,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":27885,"thread_start_us":248625,"threads_started":2,"wal-append.queue_time_us":447235}
I20260806 03:43:25.698930 3609 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.709002 767 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:25.714910 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.715652 770 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:25.717082 499 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.722970 768 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:25.724458 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:25.668583 3527 raft_consensus.cc:686] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:25.741461 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.746984 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.754102 669 raft_consensus.cc:2354] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:25.683560 3328 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.756033 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:25.765861 664 raft_consensus.cc:2354] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:25.756654 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:25.697445 3422 raft_consensus.cc:729] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:25.697431 3594 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.767117 727 leader_election.cc:398] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:25.767122 770 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:25.777757 727 leader_election.cc:401] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:43:25.707862 3178 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:25.778440 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.681124 3444 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260806 03:43:25.681094 3419 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260806 03:43:25.737569 3450 raft_consensus.cc:2741] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260806 03:43:25.680737 3367 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.070s
I20260806 03:43:25.800024 727 leader_election.cc:398] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:25.787573 665 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:25.800410 727 leader_election.cc:401] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260806 03:43:25.801729 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 94481us
W20260806 03:43:25.787109 3523 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.778s user 0.013s sys 0.000s
I20260806 03:43:25.803879 614 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:25.806640 661 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:25.763034 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:25.782943 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:25.819047 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:25.817764 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 65811.8us
I20260806 03:43:25.824831 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:25.826267 3651 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:25.807482 3610 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:25.827749 3609 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:25.786506 3637 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:25.828249 3606 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:25.829861 3394 consensus_queue.cc:1046] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:25.831166 3635 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:25.831714 1624 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:25.831086 3422 consensus_queue.cc:260] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.832077 3627 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.790563 3328 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.863565 3596 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.803224 3507 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:25.836331 548 raft_consensus.cc:2379] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:25.838572 3523 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:25.839493 3589 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.629s user 0.006s sys 0.006s
W20260806 03:43:25.844192 3639 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260806 03:43:25.846091 3641 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260806 03:43:25.800933 3223 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:25.854184 661 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 03:43:25.929495 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 110540us
I20260806 03:43:25.936626 547 raft_consensus.cc:2396] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:25.961170 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:25.796276 3178 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.835002 1665 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:25.863781 3453 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:25.928779 2936 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:26.093570 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:25.981634 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 177851us
I20260806 03:43:25.965582 3635 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:25.962582 3609 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:25.961170 3584 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:25.961604 3489 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:26.004879 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.947964 3606 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:26.004496 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:25.948575 3454 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:26.002350 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 294011us
I20260806 03:43:26.099613 563 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:26.096053 665 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:25.964816 3417 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:26.114948 564 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:26.115238 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 283612us
I20260806 03:43:26.122557 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:26.173782 661 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:26.175557 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:26.176393 558 raft_consensus.cc:2362] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3. Re-sending same reply.
I20260806 03:43:26.177443 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:26.178682 558 raft_consensus.cc:2379] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:26.180246 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:26.180971 558 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:26.125367 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:26.108177 3367 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.201782 3367 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.095335 3518 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:26.200750 563 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:26.211274 564 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:26.183386 3606 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:25.948771 3527 consensus_queue.cc:237] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.111052 3589 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:26.216823 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 118450us
I20260806 03:43:26.113588 3635 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:26.214434 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:26.224467 614 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:26.121237 3584 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:25.938784 3651 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:25.976874 3507 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:25.979429 3596 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:26.236266 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 108841us
I20260806 03:43:26.115995 3610 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.254058 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:26.098868 3429 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:26.222879 2872 consensus_meta.cc:287] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.569s user 0.012s sys 0.000s
I20260806 03:43:25.970733 3627 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:26.224355 3489 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:26.110478 3453 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:26.244674 542 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:26.236229 3584 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:26.182183 3417 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:26.233290 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:26.236371 3589 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.191535 3454 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:26.665194 3662 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.346s user 0.000s sys 0.001s
I20260806 03:43:26.553310 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:26.554773 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2054) took 1986 ms. Trace:
I20260806 03:43:26.736656 775 rpcz_store.cc:276] 0806 03:43:24.568414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.584046 (+ 15632us) service_pool.cc:227] Handling call
0806 03:43:26.554526 (+1970480us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b54001022d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:26.554708 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1814877952}
I20260806 03:43:26.630795 3178 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:26.721889 3526 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:26.781446 3526 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } attrs { replace: true } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:26.286402 3652 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:26.654794 3540 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:26.317588 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:26.341183 3669 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:26.396644 3518 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:26.711344 564 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:26.236456 3653 raft_consensus.cc:3032] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:26.299348 3645 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:26.386742 3649 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.720389 3627 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:26.637879 3394 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.710649 558 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:26.864714 3507 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:26.770177 3676 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.351s user 0.000s sys 0.001s
I20260806 03:43:26.770295 3631 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 03:43:26.778610 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 534) took 1295 ms. Trace:
I20260806 03:43:26.925052 1668 rpcz_store.cc:276] 0806 03:43:25.483534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.483843 (+ 309us) service_pool.cc:227] Handling call
0806 03:43:26.553069 (+1069226us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:26.778499 (+225430us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43502336}
W20260806 03:43:26.343067 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 172222us
I20260806 03:43:26.926828 3399 raft_consensus.cc:3032] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:26.927364 3399 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:26.928599 3399 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.929738 3454 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:26.773900 3589 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:26.857717 615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.016s sys 0.017s
W20260806 03:43:26.934929 615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s user 0.016s sys 0.017s
I20260806 03:43:26.307960 3610 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.598284 770 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 03:43:26.866884 728 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.006s sys 0.007s
W20260806 03:43:26.942571 728 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.881s user 0.007s sys 0.007s
W20260806 03:43:26.942931 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 909239us
I20260806 03:43:26.545724 3328 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:26.869182 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.010s sys 0.024s
W20260806 03:43:26.962788 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.010s sys 0.024s
I20260806 03:43:26.885244 3670 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:26.898938 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 647) took 1311 ms. Trace:
I20260806 03:43:26.868904 3540 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:26.710944 1671 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:26.827499 3489 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:26.921630 3526 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:26.936705 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:26.898859 3649 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:26.880511 3653 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:26.941869 3429 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:26.976576 1668 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:26.942183 2936 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.835s user 0.014s sys 0.001s
I20260806 03:43:26.934962 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 315) took 1099 ms. Trace:
I20260806 03:43:26.968479 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:26.990983 3648 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.941442 3627 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:26.996966 3610 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:26.970624 3507 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.970868 3596 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:26.979192 3540 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:26.974627 3589 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.016299 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1202) took 1681 ms. Trace:
I20260806 03:43:26.985826 3526 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.977695 3489 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:27.000917 775 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:26.973582 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:26.888020 3669 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.935909 3394 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.037317 564 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:26.963981 3670 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.071697 772 rpcz_store.cc:276] 0806 03:43:25.334619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.335255 (+ 636us) service_pool.cc:227] Handling call
0806 03:43:27.016121 (+1680866us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b5400101dd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.016235 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175582080}
I20260806 03:43:26.901486 3645 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:27.084884 3665 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.993944 3178 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:26.969668 1658 rpcz_store.cc:276] 0806 03:43:25.587705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.589347 (+ 1642us) service_pool.cc:227] Handling call
0806 03:43:25.650801 (+ 61454us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:25.680529 (+ 29728us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:25.680566 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:25.736944 (+ 56378us) log.cc:842] Serialized 26 byte log entry
0806 03:43:25.963364 (+226420us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:25.963444 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:25.963549 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:26.898522 (+934973us) raft_consensus.cc:1671] finished
0806 03:43:26.898589 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:26.898850 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35397,"raft.queue_time_us":17685,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":7995,"thread_start_us":200704,"threads_started":1,"wal-append.queue_time_us":418840}
I20260806 03:43:27.079954 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.123112 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 316) took 1301 ms. Trace:
I20260806 03:43:27.123541 1668 rpcz_store.cc:276] 0806 03:43:25.821981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.007685 (+1185704us) service_pool.cc:227] Handling call
0806 03:43:27.123031 (+115346us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:27.025650 3523 consensus_meta.cc:287] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.036s user 0.016s sys 0.000s
I20260806 03:43:27.005944 2936 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:27.029645 3234 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.075011 3610 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.130445 3178 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:26.998834 1663 rpcz_store.cc:276] 0806 03:43:25.817048 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.234178 (+417130us) service_pool.cc:227] Handling call
0806 03:43:26.916220 (+682042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:27.170267 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:27.173655 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 317) took 1351 ms. Trace:
I20260806 03:43:27.181643 1668 rpcz_store.cc:276] 0806 03:43:25.822201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.134147 (+1311946us) service_pool.cc:227] Handling call
0806 03:43:27.173577 (+ 39430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:27.126222 3651 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:27.138417 3523 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:27.018800 3627 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.032626 3507 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.030644 3648 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.089265 3589 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.092941 3526 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.215561 3357 raft_consensus.cc:3032] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:27.216085 3394 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:27.216516 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:27.220682 769 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1205) took 1862 ms (client timeout 2402 ms). Trace:
W20260806 03:43:27.221134 769 rpcz_store.cc:269] 0806 03:43:25.358235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.358633 (+ 398us) service_pool.cc:227] Handling call
0806 03:43:27.216346 (+1857713us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:27.220613 (+ 4267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6763392}
W20260806 03:43:27.217430 667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1650) took 2253 ms (client timeout 2084 ms). Trace:
W20260806 03:43:27.218411 3584 consensus_meta.cc:287] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.529s user 0.006s sys 0.007s
I20260806 03:43:27.212795 775 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:27.200179 3453 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:27.225486 3645 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:27.222750 667 rpcz_store.cc:269] 0806 03:43:24.964338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.964580 (+ 242us) service_pool.cc:227] Handling call
0806 03:43:27.217215 (+2252635us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54002c0954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.217349 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1586499968}
I20260806 03:43:27.125581 3665 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.229315 3584 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.232087 3648 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:27.237360 3584 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:27.238219 3422 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:27.243016 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.243938 1663 raft_consensus.cc:1799] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Handling vote request from an unknown peer 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:27.245023 1663 raft_consensus.cc:2396] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 03:43:27.229941 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.247646 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.167500 3234 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.252645 3526 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:27.171937 3610 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.256624 3610 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:27.224229 3357 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:27.249006 1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 533) took 1765 ms (client timeout 1963 ms). Trace:
I20260806 03:43:27.191782 3653 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.272433 1658 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:27.228629 3178 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:27.276921 3507 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:27.216496 3670 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:27.250233 3499 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:27.256628 3589 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:27.294252 671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.321s user 0.006s sys 0.059s
W20260806 03:43:27.299160 671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.326s user 0.006s sys 0.059s
I20260806 03:43:27.276945 3653 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:27.300814 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.025s sys 0.040s
I20260806 03:43:27.274544 3669 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:27.300323 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1657) took 1553 ms. Trace:
W20260806 03:43:27.305344 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.039s user 0.025s sys 0.040s
W20260806 03:43:27.296007 3635 consensus_meta.cc:287] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.048s user 0.009s sys 0.009s
I20260806 03:43:27.291862 3610 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.275260 3615 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:27.301844 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.118s user 0.021s sys 0.043s
W20260806 03:43:27.322772 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.022s sys 0.043s
W20260806 03:43:27.323246 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.20072e+06us
W20260806 03:43:27.333127 501 connection.cc:232] RPC call timeout handler was delayed by -1.0524215339682996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
I20260806 03:43:27.290642 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.318297 671 rpcz_store.cc:276] 0806 03:43:25.747380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.748027 (+ 647us) service_pool.cc:227] Handling call
0806 03:43:25.768237 (+ 20210us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:25.768528 (+ 291us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:43:25.972958 (+204430us) log.cc:842] Serialized 14 byte log entry
0806 03:43:27.299796 (+1326838us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:27.299868 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:27.299933 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:27.300023 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:27.300249 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":354555,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":1501818,"threads_started":2}
W20260806 03:43:27.342703 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.147s user 0.011s sys 0.054s
W20260806 03:43:27.327631 3504 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.016s sys 0.051s
I20260806 03:43:27.351230 3234 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:27.275087 3357 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:27.365612 546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1098) took 2562 ms (client timeout 1993 ms). Trace:
W20260806 03:43:27.539170 546 rpcz_store.cc:269] 0806 03:43:24.803519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.803734 (+ 215us) service_pool.cc:227] Handling call
0806 03:43:27.365246 (+2561512us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400087354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.365479 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921696512}
I20260806 03:43:27.368745 1658 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 03:43:27.272053 1660 rpcz_store.cc:269] 0806 03:43:25.483192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.486556 (+ 3364us) service_pool.cc:227] Handling call
0806 03:43:27.247413 (+1760857us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:27.248903 (+ 1490us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":59376384}
I20260806 03:43:27.302412 3499 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.374742 3394 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:27.378810 3651 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:27.389004 3524 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.392938 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 532) took 1910 ms. Trace:
I20260806 03:43:27.400065 3174 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:27.408908 728 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.963s (SENT) [suppressed 12 similar messages]
W20260806 03:43:27.411149 1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 531) took 3053 ms (client timeout 1723 ms). Trace:
I20260806 03:43:27.345471 3635 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.421370 3627 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:27.429270 3646 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 1.590s user 0.003s sys 0.008s
I20260806 03:43:27.345894 3610 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:27.512956 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.317s user 0.013s sys 0.055s
W20260806 03:43:27.467124 3606 consensus_meta.cc:287] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.239s user 0.009s sys 0.008s
I20260806 03:43:27.558065 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:27.670981 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.47501e+06us
W20260806 03:43:27.557686 729 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.421s user 0.027s sys 0.054s
I20260806 03:43:27.538451 3533 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:27.673192 729 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.537s user 0.028s sys 0.054s
I20260806 03:43:27.470711 3677 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:27.558151 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.309s user 0.028s sys 0.073s
I20260806 03:43:27.563143 3655 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.564707 3680 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:27.512526 3223 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.306s user 0.020s sys 0.099s
W20260806 03:43:27.512526 3504 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.194s user 0.017s sys 0.051s
W20260806 03:43:27.567045 1625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.629s user 0.017s sys 0.039s
W20260806 03:43:27.467478 3586 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.618s user 0.018s sys 0.098s
W20260806 03:43:27.569975 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 161085us
W20260806 03:43:27.569806 1671 rpcz_store.cc:269] 0806 03:43:24.353642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.353768 (+ 126us) service_pool.cc:227] Handling call
0806 03:43:24.704753 (+350985us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:27.407031 (+2702278us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540067cfd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.407187 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3661633024}
I20260806 03:43:27.496505 3692 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.503756 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:27.568683 1670 rpcz_store.cc:276] 0806 03:43:25.482776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.482928 (+ 152us) service_pool.cc:227] Handling call
0806 03:43:27.392004 (+1909076us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:27.392276 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:27.392529 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:27.392561 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:27.392612 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:27.392646 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:27.392716 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:27.392863 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41522432}
W20260806 03:43:27.579723 776 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2044) took 3112 ms (client timeout 2009 ms). Trace:
I20260806 03:43:27.584311 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:27.512223 3665 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:27.556977 3594 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:27.585525 561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1743) took 2611 ms (client timeout 3221 ms). Trace:
W20260806 03:43:27.668716 1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 535) took 1996 ms (client timeout 1850 ms). Trace:
W20260806 03:43:27.674834 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.426s user 0.028s sys 0.073s
W20260806 03:43:27.684321 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.066s sys 0.090s
I20260806 03:43:27.566138 3357 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:27.736557 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:27.736768 768 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 03:43:27.566463 3265 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.587s user 0.043s sys 0.044s
W20260806 03:43:27.670053 3518 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.007s sys 0.006s
I20260806 03:43:27.574069 3442 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:27.542287 3540 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.568531 3394 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.584583 3524 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:27.683045 3489 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.556s user 0.012s sys 0.003s
I20260806 03:43:27.585399 3432 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:27.578595 3419 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.035530 3527 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.316s user 0.021s sys 0.073s
W20260806 03:43:28.053108 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.108s user 0.016s sys 0.045s
I20260806 03:43:27.584298 3174 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.123175 3518 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.211s user 0.007s sys 0.010s
W20260806 03:43:27.760689 3223 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.577s user 0.021s sys 0.100s
I20260806 03:43:27.585234 3615 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.056344 561 rpcz_store.cc:269] 0806 03:43:24.973806 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.990555 (+ 16749us) service_pool.cc:227] Handling call
0806 03:43:27.578809 (+2588254us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540031fbd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.585452 (+ 6643us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2514818816}
W20260806 03:43:28.057376 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.762s user 0.092s sys 0.113s
I20260806 03:43:27.728669 3677 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.058480 1625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.130s user 0.023s sys 0.040s
I20260806 03:43:28.058915 3692 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.059979 776 rpcz_store.cc:269] 0806 03:43:24.467199 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.467385 (+ 186us) service_pool.cc:227] Handling call
0806 03:43:27.579517 (+3112132us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b5400126954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:27.579646 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2223347456}
I20260806 03:43:28.036805 771 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 03:43:28.053001 544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.388s user 0.022s sys 0.053s
W20260806 03:43:28.200538 544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.535s user 0.022s sys 0.053s
I20260806 03:43:27.683053 3507 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.040527 3526 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:27.669801 3611 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.062647 546 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:27.685376 3606 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:27.729825 3533 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.977349 3680 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:27.759802 772 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:28.003482 766 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 03:43:27.984023 769 raft_consensus.cc:1261] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:27.902458 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.236258 767 raft_consensus.cc:2379] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:27.973312 3655 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.068765 615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.025s sys 0.039s
W20260806 03:43:28.085295 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.002s sys 0.071s
W20260806 03:43:28.240319 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.277s user 0.002s sys 0.071s
W20260806 03:43:28.240075 615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.294s user 0.026s sys 0.039s
W20260806 03:43:28.243307 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 1.29736e+06us
I20260806 03:43:27.577960 3651 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.126775 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.960s user 0.035s sys 0.139s
W20260806 03:43:28.264493 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.098s user 0.035s sys 0.139s
I20260806 03:43:28.265571 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:28.116573 3422 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:28.067972 3523 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.078477 3265 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.103s user 0.048s sys 0.045s
W20260806 03:43:28.055581 3586 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.206s user 0.023s sys 0.111s
W20260806 03:43:28.188974 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.253s user 0.017s sys 0.055s
I20260806 03:43:28.110553 3589 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:28.276885 3589 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.281260 3589 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.111299 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 652) took 1161 ms. Trace:
I20260806 03:43:28.282444 1660 rpcz_store.cc:276] 0806 03:43:26.928302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.677806 (+749504us) service_pool.cc:227] Handling call
0806 03:43:27.902533 (+224727us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:27.902880 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:27.986603 (+ 83723us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:27.992830 (+ 6227us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:28.036667 (+ 43837us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.042088 (+ 5421us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:28.089783 (+ 47695us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:28.291018 546 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 03:43:28.291474 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:28.292655 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.292972 542 raft_consensus.cc:2354] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:28.296486 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.297417 558 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:28.297995 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:28.299767 542 raft_consensus.cc:2354] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:28.188246 1622 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:28.199090 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2145) took 2614 ms. Trace:
I20260806 03:43:28.062165 1665 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:28.190830 3489 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.204258 618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.327s user 0.011s sys 0.038s
I20260806 03:43:28.191329 3518 raft_consensus.cc:686] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:28.189478 3527 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.470s user 0.021s sys 0.084s
W20260806 03:43:28.232800 730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.566s user 0.047s sys 0.072s
W20260806 03:43:28.352185 730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.685s user 0.047s sys 0.073s
I20260806 03:43:28.253742 3499 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.192482 3698 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:28.204516 3627 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.197526 3223 raft_consensus.cc:686] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:28.229529 3485 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.018s user 0.043s sys 0.065s
I20260806 03:43:28.369145 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:28.209012 3695 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.257544 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.292s user 0.006s sys 0.012s
W20260806 03:43:28.371985 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.410s user 0.007s sys 0.012s
I20260806 03:43:28.296080 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:28.302471 615 connection.cc:232] RPC call timeout handler was delayed by -1.3403854498174042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
I20260806 03:43:28.145920 3442 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.061162 1664 rpcz_store.cc:269] 0806 03:43:25.672338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.583910 (+1911572us) service_pool.cc:214] Skipping call since client already timed out
0806 03:43:27.668587 (+ 84677us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:43:28.225984 3679 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:28.193405 3432 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.352897 3610 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:28.404109 615 connection.cc:232] RPC call timeout handler was delayed by -1.193253637989983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20260806 03:43:28.407567 615 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:28.357023 3586 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.338922 674 rpcz_store.cc:276] 0806 03:43:25.584219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.584431 (+ 212us) service_pool.cc:227] Handling call
0806 03:43:25.584765 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:25.585120 (+ 355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:25.585147 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:25.585688 (+ 541us) log.cc:842] Serialized 26 byte log entry
0806 03:43:25.749613 (+163925us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:25.749665 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:25.749731 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:27.426134 (+1676403us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:28.191780 (+765646us) raft_consensus.cc:1671] finished
0806 03:43:28.191802 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:28.199001 (+ 7199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8420,"raft.queue_time_us":3545,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":43199,"spinlock_wait_cycles":62450176,"thread_start_us":98953,"threads_started":1,"wal-append.queue_time_us":211922}
I20260806 03:43:28.192054 3419 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.348254 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 581251us
W20260806 03:43:28.348611 618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.099s user 0.013s sys 0.038s
I20260806 03:43:28.415402 3702 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:28.410462 3527 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.349852 501 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.004s sys 0.060s
I20260806 03:43:28.418424 564 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:28.418602 501 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.849s user 0.004s sys 0.060s
I20260806 03:43:28.409379 3518 consensus_queue.cc:237] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.419020 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 860011us
I20260806 03:43:28.214541 3507 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.395900 3703 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:28.420831 3703 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.421394 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.404234 3679 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.424490 3703 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.424896 3606 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.219494 3526 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.263386 3635 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.226933 3611 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.410202 3499 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:28.427196 3499 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.428531 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 312) took 2623 ms. Trace:
I20260806 03:43:28.429111 3611 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.432343 3526 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.304419 3699 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.433113 3701 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:28.433472 3699 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.434831 3638 log.cc:924] Time spent T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 2.677s user 0.000s sys 0.016s
I20260806 03:43:28.373198 546 raft_consensus.cc:2379] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:28.372359 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.41035e+06us
I20260806 03:43:28.371071 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:28.243860 3185 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.368342 3485 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.159s user 0.043s sys 0.065s
I20260806 03:43:28.389346 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:28.206771 3615 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.422147 3507 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:28.411911 561 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260806 03:43:28.337980 3681 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:28.355015 1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 314) took 2535 ms (client timeout 1578 ms). Trace:
I20260806 03:43:28.417050 3527 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.493674 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:28.497256 771 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:28.425717 3606 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:28.434136 3701 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.370000 1668 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:28.498215 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 539) took 1518 ms. Trace:
I20260806 03:43:28.498739 1668 rpcz_store.cc:276] 0806 03:43:26.979585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.200980 (+221395us) service_pool.cc:227] Handling call
0806 03:43:28.368934 (+1167954us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:28.498137 (+129203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4962176}
I20260806 03:43:28.304176 3523 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.445938 772 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:28.446717 562 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:28.425866 3432 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.457985 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.325943 3422 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.461427 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.464901 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.409148 615 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:28.470547 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.429205 1662 rpcz_store.cc:276] 0806 03:43:25.805210 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.426795 (+2621585us) service_pool.cc:227] Handling call
0806 03:43:28.427525 (+ 730us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:28.427908 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:28.427985 (+ 77us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:28.428087 (+ 102us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:28.428152 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.428254 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:28.428449 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:28.493649 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.380643 3627 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.531857 767 raft_consensus.cc:2379] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:28.533645 562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1104) took 2530 ms (client timeout 1937 ms). Trace:
W20260806 03:43:28.536093 562 rpcz_store.cc:269] 0806 03:43:26.003602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.003771 (+ 169us) service_pool.cc:227] Handling call
0806 03:43:28.370870 (+2367099us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b5800003398. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:28.533567 (+162697us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18265984}
I20260806 03:43:28.535053 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.537364 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.537551 773 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:28.537696 730 connection.cc:232] RPC call timeout handler was delayed by -2.0943428270984441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260806 03:43:28.538044 767 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:28.460098 3709 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:28.363281 3698 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.467878 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.472775 3419 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.471424 3710 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.470481 3635 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:28.471666 3702 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:28.456231 3553 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
W20260806 03:43:28.546442 775 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.014s sys 0.087s
W20260806 03:43:28.579200 775 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.678s user 0.014s sys 0.088s
I20260806 03:43:28.563599 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.451679 3586 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.495481 3394 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:28.545032 3540 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.529635 3694 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.580422 769 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:28.471679 3689 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:28.468146 3185 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.534639 3524 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.610098 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.567270 542 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:28.641929 548 raft_consensus.cc:2362] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3. Re-sending same reply.
W20260806 03:43:28.642751 548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1114) took 2616 ms (client timeout 1599 ms). Trace:
I20260806 03:43:28.642974 3679 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:28.643324 548 rpcz_store.cc:269] 0806 03:43:26.026077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.079192 (+ 53115us) service_pool.cc:227] Handling call
0806 03:43:28.609874 (+2530682us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b5800003398. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:28.642641 (+ 32767us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119531392}
I20260806 03:43:28.645187 3692 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.645467 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.577272 771 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:28.579751 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.565387 546 raft_consensus.cc:2379] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:28.589282 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.591629 3711 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:28.592530 3357 consensus_meta.cc:287] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.537s user 0.012s sys 0.012s
I20260806 03:43:28.648294 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.592655 3702 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.582338 3635 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.593832 3394 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:28.562932 3713 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.596294 564 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:28.597925 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.600937 3527 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.666329 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.601511 3698 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:28.492179 1665 rpcz_store.cc:269] 0806 03:43:25.816778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.228289 (+411511us) service_pool.cc:227] Handling call
0806 03:43:27.503424 (+1275135us) spinlock_profiling.cc:243] Waited 7.42 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:28.351697 (+848273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17072384}
I20260806 03:43:28.602716 3499 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:28.603211 3704 log.cc:924] Time spent T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260806 03:43:28.604512 3580 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:28.604772 3615 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.606834 3442 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.547238 3707 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:28.608656 3655 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.608970 3701 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:28.610689 1657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 297) took 5072 ms (client timeout 2139 ms). Trace:
I20260806 03:43:28.567484 3526 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.611886 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2059) took 2513 ms. Trace:
I20260806 03:43:28.755254 770 rpcz_store.cc:276] 0806 03:43:26.098167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.098321 (+ 154us) service_pool.cc:227] Handling call
0806 03:43:26.554702 (+456381us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:26.943828 (+389126us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.611650 (+1667822us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54001022d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:28.611822 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":856659712}
I20260806 03:43:28.548584 3507 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.412535 3695 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.566982 3533 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.551105 3174 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.637063 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:28.657676 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.661087 772 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:28.539728 3606 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.529235 3223 consensus_queue.cc:237] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.610772 3699 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:28.642899 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.706897 3589 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.708436 766 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:28.671802 3586 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.668977 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.669673 3185 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:28.672996 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.026s sys 0.019s
W20260806 03:43:28.808897 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.728s user 0.026s sys 0.024s
I20260806 03:43:28.667626 3394 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:28.582991 3710 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.673552 3713 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.708025 3680 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:28.650467 3653 consensus_meta.cc:287] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.325s user 0.011s sys 0.005s
W20260806 03:43:28.669901 3688 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.606s user 0.000s sys 0.001s
I20260806 03:43:28.667168 561 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:28.646337 548 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:28.671967 3635 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:28.497526 3681 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.728399 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 486278us
I20260806 03:43:28.747641 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:28.709492 3422 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:28.758391 3504 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.002s sys 0.017s
I20260806 03:43:28.730217 3432 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:28.779531 3442 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.772061 3265 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.777688 3490 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:28.814801 767 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:28.762665 3489 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.817524 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:28.819833 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.763082 3526 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.825192 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:28.854801 773 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:28.856092 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.857409 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.826179 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:28.826726 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:28.858659 772 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:28.859062 614 connection.cc:232] RPC call timeout handler was delayed by -1.6467793621122837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20260806 03:43:28.823925 3730 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:28.773447 3507 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.831250 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 643126us
I20260806 03:43:28.822115 3518 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.833308 3727 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.833324 1624 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:28.834108 3734 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.835968 558 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:28.835896 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:28.836661 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 427594us
I20260806 03:43:28.837007 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.667982 3689 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:28.817168 3394 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:28.822006 3174 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:28.840553 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 365338us
W20260806 03:43:28.821408 3594 consensus_meta.cc:287] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.758s user 0.007s sys 0.007s
I20260806 03:43:28.756244 3707 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.820622 3602 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:28.842839 661 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:28.819897 3698 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:28.754926 1657 rpcz_store.cc:269] 0806 03:43:23.538361 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:23.543067 (+ 4706us) service_pool.cc:227] Handling call
0806 03:43:23.908175 (+365108us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:28.582220 (+4674045us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b54004f4cd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:28.610495 (+ 28275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876878464}
I20260806 03:43:28.847509 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.793018 3533 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.846057 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.848254 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.850445 671 raft_consensus.cc:1261] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:28.819394 542 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:28.822196 3615 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:28.822646 3699 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:28.852802 674 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:28.806967 3589 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:28.859508 614 connection.cc:232] RPC call timeout handler was delayed by -1.6031929128803313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
W20260806 03:43:28.847005 3504 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.003s sys 0.017s
I20260806 03:43:28.832439 3627 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.825881 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2060) took 1091 ms. Trace:
I20260806 03:43:28.839002 3499 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.858084 665 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:28.870270 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:28.851359 3490 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:28.763731 3694 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.137202 667 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:29.138351 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.139218 668 raft_consensus.cc:2396] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 03:43:29.138504 614 connection.cc:232] RPC call timeout handler was delayed by -1.6013929129112512s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
I20260806 03:43:28.906482 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:29.143688 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 310427us
I20260806 03:43:29.145112 663 raft_consensus.cc:1799] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Handling vote request from an unknown peer 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:29.145852 663 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:29.136988 775 rpcz_store.cc:276] 0806 03:43:27.733992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.737032 (+ 3040us) service_pool.cc:227] Handling call
0806 03:43:27.737335 (+ 303us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:27.900970 (+163635us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:28.579638 (+678668us) log.cc:842] Serialized 14 byte log entry
0806 03:43:28.825458 (+245820us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:28.825510 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:28.825558 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.825618 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:28.825742 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":138360,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":91,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":9066,"thread_start_us":923523,"threads_started":2,"wal-append.queue_time_us":290508}
W20260806 03:43:28.919955 547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1091) took 4150 ms (client timeout 2423 ms). Trace:
I20260806 03:43:29.188896 669 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:29.190346 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.191309 669 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:29.192430 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.867511 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:29.272775 615 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:29.201078 1624 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:29.274046 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 73060.7us
I20260806 03:43:28.861943 3365 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:28.855070 3265 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.851838 3442 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.850616 3432 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.202971 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.895936 773 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:29.187904 547 rpcz_store.cc:269] 0806 03:43:24.769392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:24.770155 (+ 763us) service_pool.cc:227] Handling call
0806 03:43:28.919731 (+4149576us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54002b1454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:28.919877 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2596098816}
I20260806 03:43:28.872944 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.213161 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:29.213400 1622 connection.cc:232] RPC call timeout handler was delayed by -1.6038166300859302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260806 03:43:29.201944 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61864.7us
I20260806 03:43:29.380338 3694 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.393759 3520 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:29.823882 3365 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.825102 3520 raft_consensus.cc:686] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:29.381438 3689 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.387764 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:29.827087 3365 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:29.826469 3520 consensus_queue.cc:237] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.327013 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.867264 665 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:29.868611 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.869659 665 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:29.487658 3744 consensus_queue.cc:1046] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.383s
I20260806 03:43:29.870335 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2162) took 1008 ms. Trace:
I20260806 03:43:29.871023 665 rpcz_store.cc:276] 0806 03:43:28.861569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.868228 (+1006659us) service_pool.cc:227] Handling call
0806 03:43:29.870245 (+ 2017us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.871742 3689 raft_consensus.cc:3032] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:29.872301 3689 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:29.872546 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.874439 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.809885 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:29.884833 669 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:29.885063 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2156) took 1034 ms. Trace:
W20260806 03:43:29.312088 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 405692us
I20260806 03:43:29.211491 3561 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:29.888126 3561 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.890980 561 raft_consensus.cc:2396] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:29.414947 3679 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.436s
I20260806 03:43:29.328886 3702 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:29.893132 3702 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.895969 3702 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.830106 3655 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:29.383368 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.898783 663 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:29.899359 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1665) took 1051 ms. Trace:
I20260806 03:43:29.899751 663 rpcz_store.cc:276] 0806 03:43:28.848144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.201860 (+353716us) service_pool.cc:227] Handling call
0806 03:43:29.899279 (+697419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.901000 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.901829 663 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:29.831089 3747 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:29.832005 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:29.907534 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 75614.5us
I20260806 03:43:29.908021 3747 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:29.908253 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:29.908612 3747 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.908039 3561 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.832203 3594 consensus_queue.cc:1035] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:29.832633 3694 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:29.910787 3561 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:29.910972 3694 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:29.911089 3655 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:29.911393 3561 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:29.912173 3561 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:29.393610 3671 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:29.230420 3418 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:29.919159 3418 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.918793 3671 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.923273 3418 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:29.923657 3671 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.925448 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1675) took 1020 ms. Trace:
I20260806 03:43:29.925840 663 rpcz_store.cc:276] 0806 03:43:28.905195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.900576 (+995381us) service_pool.cc:227] Handling call
0806 03:43:29.902356 (+ 1780us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.223474 3698 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.398769 558 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:29.382601 2872 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:43:29.828745 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 556075us
I20260806 03:43:29.925987 3747 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:29.939100 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.939584 615 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:29.941300 3698 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:29.000454 3606 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:29.517028 3677 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:29.272432 3499 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:28.871531 3711 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:29.750903 3707 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:28.902853 3518 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.807739 3589 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:29.011209 3713 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.828645 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.553413 3490 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.234050 3361 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.611s
I20260806 03:43:29.962915 3490 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:29.327792 3734 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.967090 3734 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:28.937117 3591 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:29.967703 3591 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:29.968580 3591 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.014930 3523 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:29.285871 3627 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.992411 3681 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.217974 3699 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:28.904225 3733 log.cc:924] Time spent T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260806 03:43:29.223742 3174 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.416014 3417 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.306650 3504 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:29.466562 3738 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:28.966651 3695 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:28.961349 3730 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.327591 3727 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.992475 667 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:29.215721 3234 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:28.920827 3507 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:29.856518 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:28.918955 3524 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:29.717329 3710 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.200734 3692 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:29.315444 614 connection.cc:232] RPC call timeout handler was delayed by -1.2438266659155488s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260806 03:43:29.335695 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 653) took 2354 ms. Trace:
I20260806 03:43:29.397763 3602 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.223130 3533 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.976758 3627 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.827821 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 414) took 1011 ms. Trace:
I20260806 03:43:29.873502 3689 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.274904 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.828718 3746 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:29.885576 665 rpcz_store.cc:276] 0806 03:43:28.851165 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.874075 (+1022910us) service_pool.cc:227] Handling call
0806 03:43:29.884994 (+ 10919us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.827507 661 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:29.911643 3694 raft_consensus.cc:686] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:29.886624 616 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:29.283334 3526 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:29.931692 558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1767) took 1582 ms (client timeout 2106 ms). Trace:
W20260806 03:43:30.013664 558 rpcz_store.cc:269] 0806 03:43:28.349422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.747268 (+397846us) service_pool.cc:227] Handling call
0806 03:43:29.931618 (+1184350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.014621 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.015677 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.016666 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.017196 558 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.017256 661 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:30.017866 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1668) took 1165 ms. Trace:
I20260806 03:43:29.835662 546 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:30.018208 661 rpcz_store.cc:276] 0806 03:43:28.852570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.204512 (+351942us) service_pool.cc:227] Handling call
0806 03:43:30.017807 (+813295us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.018582 3444 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:30.019177 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.019613 3417 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:30.019963 661 raft_consensus.cc:2396] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:30.020354 2911 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:30.020231 3417 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.020735 2911 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:30.020815 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:29.939877 663 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:30.021724 2911 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.022703 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1679) took 1116 ms. Trace:
I20260806 03:43:29.838624 3265 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:29.929955 669 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.022811 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.023578 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2161) took 1162 ms. Trace:
I20260806 03:43:30.023732 3699 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.023947 669 rpcz_store.cc:276] 0806 03:43:28.861323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.809474 (+948151us) service_pool.cc:227] Handling call
0806 03:43:30.023522 (+214048us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.024628 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.025207 3533 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.025710 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2158) took 1171 ms. Trace:
I20260806 03:43:30.026317 661 rpcz_store.cc:276] 0806 03:43:28.854462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.024302 (+1169840us) service_pool.cc:227] Handling call
0806 03:43:30.025635 (+ 1333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.981115 3591 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.011454 3507 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:30.027123 3694 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.027932 3507 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.028846 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.029778 1657 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:30.002943 3524 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:29.984381 3647 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.613s user 0.016s sys 0.054s
W20260806 03:43:29.987699 728 leader_election.cc:334] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.723s (SENT)
I20260806 03:43:29.997488 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1671) took 1135 ms. Trace:
W20260806 03:43:30.005707 3735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s user 0.020s sys 0.021s
W20260806 03:43:30.000599 729 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.016s sys 0.046s
I20260806 03:43:29.983592 3504 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:30.002439 3604 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.166s user 0.041s sys 0.079s
I20260806 03:43:29.961457 3711 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:29.211915 3615 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:29.992617 3738 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.094121 3738 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.094437 1670 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:30.095880 3727 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.006337 773 rpcz_store.cc:276] 0806 03:43:28.816739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.817259 (+ 520us) service_pool.cc:227] Handling call
0806 03:43:29.827753 (+1010494us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:29.991613 3442 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.008270 498 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.009822 671 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:29.997128 3234 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.098654 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1676) took 1193 ms. Trace:
I20260806 03:43:30.019556 3444 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:30.020577 3174 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.024833 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:30.103921 3394 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.798s user 0.011s sys 0.000s
I20260806 03:43:30.104954 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.104650 3394 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.105602 669 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:29.389760 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:29.963884 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2163) took 1097 ms. Trace:
I20260806 03:43:30.106735 659 rpcz_store.cc:276] 0806 03:43:28.867618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.828345 (+960727us) service_pool.cc:227] Handling call
0806 03:43:29.963801 (+135456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.107174 3615 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.107646 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.108333 3627 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.108863 659 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:53596
I20260806 03:43:29.946655 2872 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.109699 2872 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.025878 3365 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:30.111079 3394 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.111786 3187 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:30.112222 3187 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.112532 3187 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:30.026461 3561 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:29.996971 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 162669us
W20260806 03:43:29.992025 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.010s sys 0.049s
I20260806 03:43:30.032047 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.009369 3583 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:30.030051 3507 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.003434 1658 rpcz_store.cc:276] 0806 03:43:26.928634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.684638 (+756004us) service_pool.cc:227] Handling call
0806 03:43:28.042697 (+358059us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:28.192949 (+150252us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:28.195319 (+ 2370us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:28.305336 (+110017us) log.cc:842] Serialized 26 byte log entry
0806 03:43:28.828747 (+523411us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:28.834743 (+ 5996us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.843663 (+ 8920us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:29.142880 (+299217us) raft_consensus.cc:1671] finished
0806 03:43:29.204901 (+ 62021us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:29.282940 (+ 78039us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":160,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":26311,"thread_start_us":425766,"threads_started":1,"wal-append.queue_time_us":643852,"wal-append.run_cpu_time_us":1913,"wal-append.run_wall_time_us":1003612}
W20260806 03:43:30.043885 3647 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.693s user 0.016s sys 0.054s
W20260806 03:43:30.044569 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 158031us
I20260806 03:43:30.005172 3692 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.004748 3710 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:30.023082 663 rpcz_store.cc:276] 0806 03:43:28.906186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.926760 (+1020574us) service_pool.cc:227] Handling call
0806 03:43:30.022645 (+ 95885us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.017990 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.072501 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.073721 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.049685 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:30.000136 614 connection.cc:232] RPC call timeout handler was delayed by -1.214020584942773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
I20260806 03:43:30.074779 728 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.074857 668 rpcz_store.cc:276] 0806 03:43:28.862705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.855956 (+993251us) service_pool.cc:227] Handling call
0806 03:43:29.997430 (+141474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.082713 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:30.082845 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.023100 2911 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:30.084582 729 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.016s sys 0.046s
I20260806 03:43:30.084913 664 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:30.074083 3746 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:30.092934 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.093367 3711 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.075470 3417 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.099005 671 rpcz_store.cc:276] 0806 03:43:28.905459 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.146985 (+241526us) service_pool.cc:227] Handling call
0806 03:43:30.098591 (+951606us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:30.099655 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 91471.5us
W20260806 03:43:30.090710 3604 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.255s user 0.041s sys 0.079s
I20260806 03:43:30.100524 3504 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.101598 3367 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:30.103156 816 heartbeater.cc:644] Failed to heartbeat to 127.30.138.126:46149 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.30.138.126:46149 timed out after 15.000s (SENT)
I20260806 03:43:30.289474 3507 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:30.109344 768 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.977s user 0.023s sys 0.071s
W20260806 03:43:30.290488 768 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.159s user 0.023s sys 0.072s
I20260806 03:43:30.291447 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2069) took 2554 ms. Trace:
I20260806 03:43:30.292613 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2159) took 1432 ms. Trace:
I20260806 03:43:30.291792 768 rpcz_store.cc:276] 0806 03:43:27.736740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.055710 (+318970us) service_pool.cc:227] Handling call
0806 03:43:28.056081 (+ 371us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:28.088331 (+ 32250us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:28.088378 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:28.131295 (+ 42917us) log.cc:842] Serialized 26 byte log entry
0806 03:43:30.290898 (+2159603us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:30.290964 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.291145 (+ 181us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:30.291196 (+ 51us) raft_consensus.cc:1671] finished
0806 03:43:30.291206 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:30.291391 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26146,"raft.queue_time_us":134253,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":12952,"thread_start_us":2158929,"threads_started":1,"wal-append.queue_time_us":705144,"wal-append.run_cpu_time_us":4787,"wal-append.run_wall_time_us":1436257}
I20260806 03:43:30.292965 663 rpcz_store.cc:276] 0806 03:43:28.860717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.264984 (+1404267us) service_pool.cc:227] Handling call
0806 03:43:30.292135 (+ 27151us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.292319 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.292343 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:30.292382 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:30.292412 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.292459 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:30.292557 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.116645 3561 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:30.257906 614 connection.cc:232] RPC call timeout handler was delayed by -1.1411868478171527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
I20260806 03:43:30.295389 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.259152 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.297237 773 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:30.259259 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:30.259485 546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1770) took 1891 ms (client timeout 2002 ms). Trace:
I20260806 03:43:30.116724 3738 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.260376 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.260551 3583 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.261475 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.261518 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2151) took 1432 ms. Trace:
I20260806 03:43:30.085039 548 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:30.122857 659 raft_consensus.cc:595] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.263587 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.122539 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.265424 668 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:30.263135 558 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.142819 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:30.267875 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.853s user 0.015s sys 0.055s
I20260806 03:43:30.117890 562 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:30.264032 1664 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:30.281920 671 raft_consensus.cc:1226] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:30.099226 766 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 399) took 1734 ms (client timeout 1716 ms). Trace:
I20260806 03:43:30.284196 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:30.143934 3518 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.288800 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.403545 771 raft_consensus.cc:2362] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6. Re-sending same reply.
I20260806 03:43:30.408308 769 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.299120 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:30.407678 3771 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260806 03:43:30.409312 772 raft_consensus.cc:2379] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:30.404168 766 rpcz_store.cc:269] 0806 03:43:28.359635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.362833 (+ 3198us) service_pool.cc:227] Handling call
0806 03:43:30.094170 (+1731337us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54002f5154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.094296 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":563474176}
I20260806 03:43:30.298765 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:30.401153 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.00636e+06us
I20260806 03:43:30.098052 3649 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.326803 663 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260806 03:43:30.400214 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2160) took 1539 ms. Trace:
I20260806 03:43:30.400279 659 raft_consensus.cc:603] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:30.400017 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 280) took 1124 ms. Trace:
I20260806 03:43:30.400291 674 rpcz_store.cc:276] 0806 03:43:28.829615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.830478 (+ 863us) service_pool.cc:227] Handling call
0806 03:43:28.830866 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:29.140177 (+309311us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.261256 (+1121079us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b540036ced4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:30.261450 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":49681792}
I20260806 03:43:30.416931 558 rpcz_store.cc:276] 0806 03:43:29.275725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.017784 (+742059us) service_pool.cc:227] Handling call
0806 03:43:30.399953 (+382169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.416651 671 rpcz_store.cc:276] 0806 03:43:28.861052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.267633 (+1406581us) service_pool.cc:227] Handling call
0806 03:43:30.281670 (+ 14037us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.399953 (+118283us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.400162 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:30.399740 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 103176us
I20260806 03:43:30.402693 775 raft_consensus.cc:2396] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:43:30.400099 546 rpcz_store.cc:269] 0806 03:43:28.368527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.657337 (+288810us) service_pool.cc:227] Handling call
0806 03:43:30.259324 (+1601987us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540030eb54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.259429 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2434444672}
I20260806 03:43:30.399608 767 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.417752 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.400695 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2157) took 1546 ms. Trace:
I20260806 03:43:30.437202 674 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:30.403903 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.438869 768 raft_consensus.cc:2396] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:43:30.288990 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 301316us
I20260806 03:43:30.439728 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.377581 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.400563 1670 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:30.353055 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.414611 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.415359 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.416512 770 raft_consensus.cc:2396] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:30.418656 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.443217 1664 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:30.420409 1624 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:30.421972 3730 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.286485 1662 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:43:30.299522 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.300477 3698 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:30.279978 3710 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.424294 3602 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.424636 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1674) took 1519 ms. Trace:
I20260806 03:43:30.425727 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.298277 3174 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:30.431224 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.431773 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1673) took 1538 ms. Trace:
I20260806 03:43:30.372622 3504 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.432133 1663 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 03:43:30.432371 561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1774) took 1601 ms (client timeout 1838 ms). Trace:
I20260806 03:43:30.432164 3694 consensus_queue.cc:1035] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:30.436056 614 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.437813 668 rpcz_store.cc:276] 0806 03:43:28.854147 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.264825 (+1410678us) service_pool.cc:227] Handling call
0806 03:43:30.265110 (+ 285us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.400436 (+135326us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.400642 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.460258 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.438704 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.407263 3399 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:30.441097 773 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.527674 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.441838 1668 raft_consensus.cc:2354] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4.
I20260806 03:43:30.442091 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.528872 1670 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:30.442116 665 raft_consensus.cc:2396] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:30.444293 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:30.445812 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 84264.1us
I20260806 03:43:30.530440 546 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:30.447038 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.448518 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.285899 3746 raft_consensus.cc:686] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:30.298027 3711 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:30.449913 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:30.450238 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 43779.4us
I20260806 03:43:30.321529 3499 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.458904 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.401862 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.458007 2911 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:30.523054 561 rpcz_store.cc:269] 0806 03:43:28.830608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.867225 (+ 36617us) service_pool.cc:227] Handling call
0806 03:43:30.432136 (+1564911us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b5400087354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.432309 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1423524096}
I20260806 03:43:30.551882 775 raft_consensus.cc:2396] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:30.524328 771 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.524003 772 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:30.536794 674 raft_consensus.cc:2354] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3.
I20260806 03:43:30.525103 663 rpcz_store.cc:276] 0806 03:43:28.879570 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.293924 (+1414354us) service_pool.cc:227] Handling call
0806 03:43:30.294290 (+ 366us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.418044 (+123754us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.418279 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.495484 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.463536 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.532967 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.626933 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:30.533901 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 97921us
I20260806 03:43:30.626154 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.535773 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:30.627952 669 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.536793 671 raft_consensus.cc:2396] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:30.530414 667 rpcz_store.cc:276] 0806 03:43:28.904887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.212801 (+307914us) service_pool.cc:227] Handling call
0806 03:43:30.424432 (+1211631us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002ce054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.424573 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378582784}
I20260806 03:43:30.629057 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.629380 614 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.630087 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.630766 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.540762 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.548115 768 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:30.631490 773 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:30.631814 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.632151 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.632429 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:30.632570 669 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:30.548897 498 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.522152 1665 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.461253 3621 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:30.630869 667 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:30.632947 773 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:30.548081 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.538959 661 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:30.631448 668 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
W20260806 03:43:30.647104 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 98262.7us
I20260806 03:43:30.652117 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.633473 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:30.634090 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.630112 1622 leader_election.cc:398] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3.
I20260806 03:43:30.653241 1622 leader_election.cc:401] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:30.552211 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1110) took 4541 ms. Trace:
I20260806 03:43:30.653959 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.654793 1622 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.635035 1664 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:30.657682 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.656536 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.660172 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.561662 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.632894 768 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.661011 547 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
W20260806 03:43:30.661465 547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 276) took 1507 ms (client timeout 1712 ms). Trace:
I20260806 03:43:30.661597 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.637145 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:30.662024 547 rpcz_store.cc:269] 0806 03:43:29.153739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.561302 (+1407563us) service_pool.cc:227] Handling call
0806 03:43:30.661411 (+100109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.654443 563 rpcz_store.cc:276] 0806 03:43:26.005772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:26.771739 (+765967us) service_pool.cc:227] Handling call
0806 03:43:27.370486 (+598747us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:27.401824 (+ 31338us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 03:43:27.401869 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:27.401924 (+ 55us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:27.401966 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:27.402054 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:30.547449 (+3145395us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400087350. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.547566 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39793,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":6286,"spinlock_wait_cycles":1375862528}
I20260806 03:43:30.660578 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:30.639118 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.663564 1670 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:30.650098 546 raft_consensus.cc:2396] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:30.652426 668 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:30.658541 772 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:30.662338 768 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:30.550875 3527 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:30.621680 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:30.669718 665 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 5
W20260806 03:43:30.559604 564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1119) took 3002 ms (client timeout 1590 ms). Trace:
I20260806 03:43:30.662884 671 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:30.665377 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:30.626662 770 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:30.561383 3234 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.668896 615 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.674222 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1773) took 1843 ms. Trace:
I20260806 03:43:30.565879 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.674779 546 rpcz_store.cc:276] 0806 03:43:28.830304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.547830 (+1717526us) service_pool.cc:227] Handling call
0806 03:43:30.674146 (+126316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.663898 614 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.676321 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.672358 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.677240 563 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:30.565555 3651 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.679114 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.679593 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.612627 3432 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.680826 1670 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:30.637884 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.676609 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.686461 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.683246 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:30.687512 561 raft_consensus.cc:2396] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:30.688314 615 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.689944 546 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:30.690701 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1772) took 1860 ms. Trace:
I20260806 03:43:30.691095 546 rpcz_store.cc:276] 0806 03:43:28.830035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.686162 (+1856127us) service_pool.cc:227] Handling call
0806 03:43:30.690633 (+ 4471us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.692214 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.693027 546 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:30.563423 3738 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.657181 3615 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:43:30.678243 564 rpcz_store.cc:269] 0806 03:43:27.557680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:27.557791 (+ 111us) service_pool.cc:227] Handling call
0806 03:43:30.559421 (+3001630us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54000cf654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.559530 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1719147392}
I20260806 03:43:30.565879 3710 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.489219 3698 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.677747 768 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.668201 3602 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:30.680333 3665 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:30.642194 3499 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.520013 3187 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.683396 614 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.577915 3711 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.694047 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:30.666971 3332 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:30.700929 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:30.733340 665 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:30.740051 547 raft_consensus.cc:2396] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:30.740311 546 raft_consensus.cc:1261] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:30.740819 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.708204 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:30.743671 665 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:30.743755 770 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.702770 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.747277 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.747931 665 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.748133 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.749769 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.711961 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.750777 564 raft_consensus.cc:1799] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Handling vote request from an unknown peer 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.751524 564 raft_consensus.cc:2396] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260806 03:43:30.752005 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1762) took 2447 ms. Trace:
I20260806 03:43:30.752578 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.672979 3694 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:30.563702 2872 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.754609 3694 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.755016 3694 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:30.756477 1660 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:30.696767 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.756671 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.757620 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.757890 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.759104 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:30.759176 548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1775) took 1928 ms (client timeout 1701 ms). Trace:
I20260806 03:43:30.758276 772 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
W20260806 03:43:30.759781 548 rpcz_store.cc:269] 0806 03:43:28.830837 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.847725 (+ 16888us) service_pool.cc:227] Handling call
0806 03:43:30.758936 (+1911211us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b540032b4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.759081 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1931844480}
I20260806 03:43:30.761701 1624 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:30.761883 548 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:30.678457 3448 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:30.776018 3448 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.776454 3448 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:30.776633 3698 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.695990 3770 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:30.696776 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.672849 3518 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:30.706188 3429 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:30.674568 3621 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:30.720893 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.725145 3710 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:30.725847 3615 raft_consensus.cc:686] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:30.726042 768 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:30.701774 3234 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
W20260806 03:43:30.728286 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 44965.3us
I20260806 03:43:30.797528 614 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:30.799038 614 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:30.699615 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.731997 3583 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:30.741182 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1123) took 2266 ms. Trace:
I20260806 03:43:30.807085 546 rpcz_store.cc:276] 0806 03:43:28.474921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.737897 (+2262976us) service_pool.cc:227] Handling call
0806 03:43:30.738278 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.740943 (+ 2665us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.741130 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.808331 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.809809 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.811206 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.698419 773 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:30.742928 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 275) took 1589 ms. Trace:
I20260806 03:43:30.818821 547 rpcz_store.cc:276] 0806 03:43:29.153378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.673604 (+1520226us) service_pool.cc:227] Handling call
0806 03:43:30.742862 (+ 69258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.756335 2872 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.694931 3715 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:30.762681 3738 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:30.760767 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.715657 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1768) took 2366 ms. Trace:
I20260806 03:43:30.764531 3332 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:30.752374 564 rpcz_store.cc:276] 0806 03:43:28.305081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.711674 (+2406593us) service_pool.cc:227] Handling call
0806 03:43:30.751934 (+ 40260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.828644 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.772766 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1744) took 5779 ms. Trace:
W20260806 03:43:30.773871 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.95076e+06us
I20260806 03:43:30.791877 3651 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:30.758169 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.785996 558 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:30.786028 775 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:30.575294 3746 consensus_queue.cc:237] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.840885 498 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.742815 501 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.802755 3770 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:30.701785 3527 raft_consensus.cc:686] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:30.792430 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.848166 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.848664 768 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:30.823717 3715 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.793290 674 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:30.793650 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.853808 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.855309 668 raft_consensus.cc:2379] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:30.722122 3665 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.713676 3432 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260806 03:43:30.710760 3618 consensus_queue.cc:1046] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260806 03:43:30.831346 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.731374 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:30.804476 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:30.927053 563 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.819521 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:30.933578 3485 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:30.934149 3485 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.937341 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.938068 767 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:30.940060 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:30.939631 3615 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.940831 767 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:30.941587 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.785236 3429 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:30.942801 3429 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.942510 3621 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.743127 3692 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:30.780251 3174 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:30.823385 546 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:30.848486 2911 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:30.848176 3419 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:30.827065 561 rpcz_store.cc:276] 0806 03:43:28.349711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.637554 (+2287843us) service_pool.cc:227] Handling call
0806 03:43:30.715593 (+ 78039us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.775305 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.829453 564 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.863969 3715 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.785116 3518 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:30.785431 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 88728.7us
I20260806 03:43:30.969831 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.846735 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.971148 772 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:30.821398 3609 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:30.923935 3602 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:30.847432 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 104748us
I20260806 03:43:30.987955 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.989593 501 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.849820 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.986184 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:30.991211 772 raft_consensus.cc:2396] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:30.727026 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.826396 3738 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:30.802007 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 273) took 1969 ms. Trace:
I20260806 03:43:30.910547 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:31.006029 501 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.006676 775 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:31.007272 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:30.854986 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:30.920851 562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 277) took 1766 ms (client timeout 1700 ms). Trace:
I20260806 03:43:31.007974 563 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:31.008177 562 rpcz_store.cc:269] 0806 03:43:29.154040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:29.203786 (+ 49746us) service_pool.cc:227] Handling call
0806 03:43:30.920539 (+1716753us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400086454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:30.920754 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25206144}
W20260806 03:43:31.008365 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 153473us
I20260806 03:43:30.921588 3490 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:30.857596 3332 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:30.939098 3485 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:30.835608 3651 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:30.948383 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:30.948702 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 129302us
I20260806 03:43:31.036405 3689 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.037580 767 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 2.
I20260806 03:43:31.043371 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.045095 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.047597 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:31.049156 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.052485 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.053180 542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1761) took 2748 ms (client timeout 2554 ms). Trace:
W20260806 03:43:31.053855 542 rpcz_store.cc:269] 0806 03:43:28.304745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.052863 (+2748118us) service_pool.cc:214] Skipping call since client already timed out
0806 03:43:31.053054 (+ 191us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:43:31.057426 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:31.058308 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:53596 (request call id 9) took 1244 ms. Trace:
I20260806 03:43:31.058871 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.952840 3403 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:31.059725 3403 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:31.060812 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 278) took 1906 ms. Trace:
I20260806 03:43:31.060664 3403 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.958163 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 274) took 1827 ms. Trace:
I20260806 03:43:31.061373 542 rpcz_store.cc:276] 0806 03:43:29.154349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.059639 (+1905290us) service_pool.cc:227] Handling call
0806 03:43:31.060080 (+ 441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:31.060342 (+ 262us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:31.060387 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:31.060450 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:31.060508 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.060589 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:31.060736 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.782842 770 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.820652 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:31.062934 664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1661) took 2216 ms (client timeout 1854 ms). Trace:
W20260806 03:43:31.063341 664 rpcz_store.cc:269] 0806 03:43:28.846745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:28.846905 (+ 160us) service_pool.cc:227] Handling call
0806 03:43:31.062765 (+2215860us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000b1b54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.062869 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2824517376}
I20260806 03:43:31.064455 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:31.065330 770 raft_consensus.cc:1261] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:43:30.964151 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:31.066764 561 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:30.829946 544 rpcz_store.cc:276] 0806 03:43:24.993166 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:25.136048 (+142882us) service_pool.cc:227] Handling call
0806 03:43:26.663281 (+1527233us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b5800003398. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:26.663558 (+ 277us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:26.663887 (+ 329us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:26.663938 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:26.664574 (+ 636us) log.cc:842] Serialized 26 byte log entry
0806 03:43:28.200923 (+1536349us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:28.222001 (+ 21078us) log.cc:842] Serialized 14 byte log entry
0806 03:43:28.222247 (+ 246us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:28.222341 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:30.770662 (+2548321us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b1800452ec0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:30.772539 (+ 1877us) raft_consensus.cc:1671] finished
0806 03:43:30.772560 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:30.772679 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10325,"prepare.queue_time_us":28858,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":3094,"raft.queue_time_us":136527,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":17317,"spinlock_wait_cycles":149819520,"thread_start_us":1535802,"threads_started":1,"wal-append.queue_time_us":1636112}
I20260806 03:43:31.062523 546 rpcz_store.cc:276] 0806 03:43:29.131303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.822689 (+1691386us) service_pool.cc:227] Handling call
0806 03:43:30.823066 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.957884 (+134818us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.958107 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:30.964424 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:30.843851 1668 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:31.071691 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.072008 770 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:31.072312 1668 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:31.073069 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.073422 770 raft_consensus.cc:1261] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 03:43:31.073887 770 pending_rounds.cc:76] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 1
I20260806 03:43:31.074108 770 pending_rounds.cc:97] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 03:43:31.074350 770 raft_consensus.cc:2866] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:30.912410 3665 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.076457 1624 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:30.846612 547 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:30.963160 2911 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.077246 3489 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:31.077685 2911 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:31.077888 3489 raft_consensus.cc:686] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:31.078892 3489 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:30.966023 3524 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.009284 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:31.083745 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.074182 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.077279 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1771) took 2247 ms. Trace:
I20260806 03:43:31.086470 547 rpcz_store.cc:276] 0806 03:43:28.829726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.823859 (+1994133us) service_pool.cc:227] Handling call
0806 03:43:30.846259 (+ 22400us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:31.077044 (+230785us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.077225 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:31.087095 1671 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:31.085605 402 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35388 (request call id 318) took 18130 ms (client timeout 15000 ms). Trace:
W20260806 03:43:31.087911 402 rpcz_store.cc:269] 0806 03:43:12.955396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:12.955573 (+ 177us) service_pool.cc:227] Handling call
0806 03:43:31.085393 (+18129820us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108208,"reported_tablets":7}
W20260806 03:43:30.949205 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 338652us
I20260806 03:43:31.011271 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:30.986949 3518 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.017004 772 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:30.981154 3609 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:31.092481 3609 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.093489 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 285) took 1225 ms. Trace:
I20260806 03:43:31.093840 558 rpcz_store.cc:276] 0806 03:43:29.868026 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.438487 (+570461us) service_pool.cc:227] Handling call
0806 03:43:31.093308 (+654821us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54002b0a54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.093430 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":602960640}
I20260806 03:43:31.096621 3365 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.097101 3365 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:31.028757 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:30.986865 3602 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.100127 3602 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:30.964391 1663 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:30.986706 2872 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:31.102953 2872 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.103365 2872 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:31.058756 659 rpcz_store.cc:276] 0806 03:43:29.813679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.108723 (+295044us) service_pool.cc:227] Handling call
0806 03:43:31.058122 (+949399us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b540016dad4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.058242 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30791936}
I20260806 03:43:31.113607 401 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51378 (request call id 178) took 8856 ms. Trace:
I20260806 03:43:31.115118 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 03:43:31.117846 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:31.119022 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:31.010685 501 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.121219 2872 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.121651 3357 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:31.122673 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 112085us
I20260806 03:43:31.124099 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.125550 3357 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.125939 3357 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.127094 501 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.128266 3599 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.128693 3599 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:31.128680 501 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.129307 3599 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.114025 401 rpcz_store.cc:276] 0806 03:43:22.271985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:22.273663 (+ 1678us) service_pool.cc:227] Handling call
0806 03:43:22.444240 (+170577us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b5400002dd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321c279d6c 00007f321c373c3e 00007f321a685fb4 00007f321a685f42 00007f321a685e71 00007f321a685dec 00007f321a6850a2 00007f321474994c 00007f3214748d86
0806 03:43:31.113537 (+8669297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:24.204046 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:24.234906 (+ 30860us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:24.234958 (+ 52us) write_op.cc:435] Acquired schema lock
0806 03:43:24.234978 (+ 20us) tablet.cc:660] Decoding operations
0806 03:43:24.289051 (+ 54073us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:24.291401 (+ 2350us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:24.293605 (+ 2204us) tablet.cc:683] Acquiring locks for 11 operations
0806 03:43:24.504813 (+211208us) tablet.cc:699] Row locks acquired
0806 03:43:24.504836 (+ 23us) write_op.cc:260] PREPARE: finished
0806 03:43:24.570233 (+ 65397us) write_op.cc:270] Start()
0806 03:43:24.586168 (+ 15935us) write_op.cc:276] Timestamp: P: 1785987804564954 usec, L: 0
0806 03:43:24.586198 (+ 30us) op_driver.cc:331] REPLICATION: starting
0806 03:43:24.796499 (+210301us) log.cc:842] Serialized 4430 byte log entry
0806 03:43:25.478329 (+681830us) op_driver.cc:444] REPLICATION: finished
0806 03:43:27.405382 (+1927053us) spinlock_profiling.cc:243] Waited 2.76 ms on lock 0x7b18002e25e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:27.405568 (+ 186us) write_op.cc:301] APPLY: starting
0806 03:43:27.405666 (+ 98us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:27.432278 (+ 26612us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b18002e25e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7799 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:27.512095 (+ 79817us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:27.538487 (+ 26392us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:28.535254 (+996767us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:28.619978 (+ 84724us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0806 03:43:28.649046 (+ 29068us) write_op.cc:312] APPLY: finished
0806 03:43:28.776353 (+127307us) log.cc:842] Serialized 83 byte log entry
0806 03:43:28.853508 (+ 77155us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:28.904341 (+ 50833us) write_op.cc:454] Released schema lock
0806 03:43:28.904421 (+ 80us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:29.230524 (+326103us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:29.275851 (+ 45327us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":53247,"reported_tablets":19,"spinlock_wait_cycles":101574400,"child_traces":[["op",{"apply.queue_time_us":274044,"apply.run_cpu_time_us":36847,"apply.run_wall_time_us":2009969,"num_ops":11,"prepare.queue_time_us":186863,"prepare.run_cpu_time_us":41035,"prepare.run_wall_time_us":791735,"raft.queue_time_us":256331,"raft.run_cpu_time_us":69628,"raft.run_wall_time_us":275046,"replication_time_us":890333,"spinlock_wait_cycles":20803840,"thread_start_us":709403,"threads_started":5,"wal-append.queue_time_us":1450642,"wal-append.run_cpu_time_us":55500,"wal-append.run_wall_time_us":2170242}]]}
I20260806 03:43:31.133744 2911 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.135953 2911 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.139493 3665 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.140976 1670 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:31.141060 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:31.142028 659 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:43:31.126526 3518 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.126320 3357 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.148936 3609 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.150944 669 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:31.154188 3518 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.111039 3365 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.157608 3365 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.158595 3596 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:31.159034 3596 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.159382 3596 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.161283 616 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:31.162171 3178 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:31.162571 3178 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:31.162920 3178 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.167783 3596 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.170193 3596 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.171486 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:31.171845 2911 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:31.172360 2911 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.185949 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:31.171535 501 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.179440 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:31.277730 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 106284us
I20260806 03:43:31.279575 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:31.010696 3738 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:31.156188 3599 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:31.013367 3616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 5.186s user 0.030s sys 0.061s
I20260806 03:43:31.275936 562 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 4
W20260806 03:43:31.284628 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 105309us
I20260806 03:43:31.014554 3698 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:31.010341 3419 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.271776 3609 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.012223 3490 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:31.177315 3178 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:31.298611 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 289252us
I20260806 03:43:31.298750 3770 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:31.015123 3618 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:31.299235 3422 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.178330 3357 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.015520 3334 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:31.282056 3328 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:31.302728 1657 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:31.358608 3490 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.304271 3711 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.304770 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:31.305492 3710 raft_consensus.cc:3032] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:31.403517 3710 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:31.361267 3770 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:31.404124 3770 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:31.305918 3504 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.773s user 0.007s sys 0.003s
I20260806 03:43:31.014725 3234 raft_consensus.cc:2741] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 03:43:31.317292 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:31.286393 728 leader_election.cc:334] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.731s (SENT)
W20260806 03:43:31.405742 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 88604.1us
W20260806 03:43:31.406039 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 119704us
W20260806 03:43:31.406754 728 leader_election.cc:334] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.850s (SENT)
I20260806 03:43:31.406958 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:31.407265 728 leader_election.cc:334] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.963s (SENT)
I20260806 03:43:31.407729 3357 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.408402 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:31.320187 501 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:31.313019 659 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:31.409425 776 raft_consensus.cc:2396] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:31.327085 3596 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:43:31.317895 3616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 5.490s user 0.034s sys 0.069s
I20260806 03:43:31.410604 3609 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:31.186276 3527 consensus_queue.cc:237] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.411047 3609 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.411327 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fcbc41710a42cd9ee5dacc5d9577fb with delete_type TABLET_DATA_TOMBSTONED (TS 0374a84d5e9f4e3c8d8287e2382379c8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:47796
W20260806 03:43:31.388594 3448 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.612s user 0.012s sys 0.000s
I20260806 03:43:31.412012 3693 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:31.412601 3693 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:31.412323 3448 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.308668 3600 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:31.339612 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.258356 3187 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.415094 3518 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:31.415470 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.359918 727 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:31.404419 3710 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.334928 562 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:31.315102 3432 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:31.366453 661 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.003s sys 0.019s
I20260806 03:43:31.376724 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.416888 3710 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.416839 3432 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:31.417338 1664 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:31.417008 3328 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:31.418470 661 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.738s user 0.003s sys 0.019s
I20260806 03:43:31.418992 3432 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:31.389716 669 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:31.033814 3651 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:31.303802 2911 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.419554 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.372192 3399 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.832s user 0.011s sys 0.001s
I20260806 03:43:31.419845 3504 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.420455 771 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:31.358683 3419 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.420817 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.421141 3419 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.421485 1663 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:31.402951 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:31.422639 3334 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:31.006044 548 rpcz_store.cc:276] 0806 03:43:28.821405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.761119 (+1939714us) service_pool.cc:227] Handling call
0806 03:43:30.761555 (+ 436us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.787481 (+ 25926us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:30.791064 (+ 3583us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432}
I20260806 03:43:31.386955 3422 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.411442 3609 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.413012 3693 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.477262 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:31.345562 615 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.000s sys 0.018s
W20260806 03:43:31.477929 615 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.776s user 0.001s sys 0.018s
W20260806 03:43:31.416230 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 56447.7us
I20260806 03:43:31.423852 1660 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:31.418212 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:31.380684 769 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.003s sys 0.022s
W20260806 03:43:31.482669 769 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.784s user 0.003s sys 0.022s
I20260806 03:43:31.410094 3365 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:31.436039 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:31.423292 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 103198us
I20260806 03:43:31.429075 1668 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:31.423084 728 leader_election.cc:334] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.572s (SENT)
W20260806 03:43:31.505542 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 82493.4us
W20260806 03:43:31.506162 728 leader_election.cc:334] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.712s (SENT)
I20260806 03:43:31.506521 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2087) took 1081 ms. Trace:
I20260806 03:43:31.507787 728 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:31.427587 498 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:31.509361 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:31.410404 3429 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.185328 3665 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.360474 3698 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:31.601243 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 84392.2us
I20260806 03:43:31.476883 3422 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.423822 3301 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 03:43:31.479166 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2077) took 1218 ms. Trace:
I20260806 03:43:31.624339 776 rpcz_store.cc:276] 0806 03:43:30.261110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.261241 (+ 131us) service_pool.cc:227] Handling call
0806 03:43:31.473695 (+1212454us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400321c54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.479101 (+ 5406us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2321493760}
I20260806 03:43:31.625557 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:31.626858 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:31.627918 776 raft_consensus.cc:2396] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:31.416553 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.478274 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 790023us
I20260806 03:43:31.482664 1664 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:31.485625 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 655) took 1454 ms. Trace:
I20260806 03:43:31.406080 3738 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.493351 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2090) took 1065 ms. Trace:
I20260806 03:43:31.646885 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:31.644004 727 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:31.492728 1662 consensus_meta.cc:287] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.003s user 0.008s sys 0.004s
I20260806 03:43:31.496250 3178 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.423461 3596 raft_consensus.cc:686] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:31.401444 3599 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.418992 3328 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:31.506875 769 rpcz_store.cc:276] 0806 03:43:30.425051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.425193 (+ 142us) service_pool.cc:227] Handling call
0806 03:43:30.425583 (+ 390us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.425988 (+ 405us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.426019 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.426446 (+ 427us) log.cc:842] Serialized 26 byte log entry
0806 03:43:30.642940 (+216494us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:31.505928 (+862988us) log.cc:842] Serialized 14 byte log entry
0806 03:43:31.506175 (+ 247us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.506277 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:31.506319 (+ 42us) raft_consensus.cc:1671] finished
0806 03:43:31.506329 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:31.506467 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":974,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"thread_start_us":1000233,"threads_started":2,"wal-append.queue_time_us":310442}
I20260806 03:43:31.424083 3357 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.509168 3365 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:31.511258 3604 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.514565 501 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:31.671794 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 157318us
W20260806 03:43:31.515374 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 87904.7us
I20260806 03:43:31.424465 3725 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:31.462865 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.658708 3450 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:31.654330 1662 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:31.473874 3602 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.423323 3711 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.094s
I20260806 03:43:31.622663 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.629140 728 leader_election.cc:334] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 2.045s (SENT)
I20260806 03:43:31.707466 1668 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:31.677227 3599 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:31.641255 667 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s user 0.016s sys 0.041s
W20260806 03:43:31.714102 667 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s user 0.017s sys 0.042s
W20260806 03:43:31.641573 665 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.018s sys 0.041s
W20260806 03:43:31.715103 665 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s user 0.018s sys 0.041s
I20260806 03:43:31.683888 3450 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:31.653563 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 149675us
I20260806 03:43:31.719357 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:31.722160 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.723897 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:31.533603 3448 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.603667 3703 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:31.576794 2872 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:31.623706 3698 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.676450 769 raft_consensus.cc:1261] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:31.601934 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:31.620318 3399 consensus_queue.cc:1046] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260806 03:43:31.585536 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:31.624673 3583 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260806 03:43:31.671119 3604 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.635458 1657 rpcz_store.cc:276] 0806 03:43:30.030927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.032434 (+ 1507us) service_pool.cc:227] Handling call
0806 03:43:31.485436 (+1453002us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400577954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.485564 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2379479296}
I20260806 03:43:31.691466 3596 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.687821 3675 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3.
I20260806 03:43:31.694221 3738 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.670917 3365 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.669442 3357 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.637207 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.795674 1670 raft_consensus.cc:2379] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:31.474574 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.649866 776 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260806 03:43:31.704953 3746 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s user 0.019s sys 0.036s
I20260806 03:43:31.680998 3600 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260806 03:43:31.730204 3399 raft_consensus.cc:3032] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:31.730273 3429 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:31.720592 3450 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:31.732303 772 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:31.748826 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.820317 773 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.008s sys 0.006s
I20260806 03:43:31.734704 771 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:31.735707 1622 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.024s sys 0.048s
W20260806 03:43:31.709030 1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 657) took 1668 ms (client timeout 2142 ms). Trace:
I20260806 03:43:31.780362 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.683429 3725 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.796502 1671 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:31.694298 3693 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.793221 1664 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:31.798980 3746 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.019s sys 0.036s
I20260806 03:43:31.803095 3504 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:31.813223 3399 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:31.737676 3328 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:31.748425 3604 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.805838 3357 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.749864 3234 raft_consensus.cc:3032] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:31.776626 3703 raft_consensus.cc:686] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:31.797504 3677 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:31.729741 3618 consensus_queue.cc:1046] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
W20260806 03:43:31.813962 3499 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.077s user 0.007s sys 0.006s
I20260806 03:43:31.825150 1668 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:31.822167 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:31.821223 3715 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.824s user 0.011s sys 0.001s
I20260806 03:43:31.833624 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.803785 3490 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:31.821350 3489 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.001s sys 0.014s
W20260806 03:43:31.817277 3403 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.020s sys 0.037s
W20260806 03:43:31.834683 3694 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.079s user 0.010s sys 0.004s
I20260806 03:43:31.837051 3328 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:31.840682 3234 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:31.849766 3499 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:31.847093 3677 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:31.851112 2911 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:31.859190 3489 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s user 0.001s sys 0.015s
W20260806 03:43:31.860467 3403 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s user 0.021s sys 0.037s
W20260806 03:43:31.866051 3332 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.012s sys 0.024s
I20260806 03:43:31.849982 3693 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.866183 3328 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:31.880406 3677 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.856376 3715 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.883117 3490 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.883438 3715 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.873399 3703 consensus_queue.cc:237] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:31.874349 3615 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.010s sys 0.006s
I20260806 03:43:31.886214 3602 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.851994 3770 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.866699 3694 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:31.874195 3621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s user 0.007s sys 0.012s
I20260806 03:43:31.680212 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:32.088426 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 408141us
I20260806 03:43:32.088941 3618 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:31.965271 563 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:31.871786 3234 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.093628 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 293) took 1269 ms. Trace:
W20260806 03:43:31.974222 664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.000s sys 0.019s
W20260806 03:43:32.094748 664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s user 0.000s sys 0.020s
W20260806 03:43:31.974226 663 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.960s user 0.006s sys 0.008s
W20260806 03:43:32.096123 663 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s user 0.007s sys 0.008s
I20260806 03:43:31.975008 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2188) took 1230 ms. Trace:
I20260806 03:43:32.096877 665 rpcz_store.cc:276] 0806 03:43:30.744009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.772764 (+ 28755us) service_pool.cc:227] Handling call
0806 03:43:30.773094 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.773470 (+ 376us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.773496 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.773961 (+ 465us) log.cc:842] Serialized 26 byte log entry
0806 03:43:31.715553 (+941592us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:31.715608 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.715692 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:31.974156 (+258464us) raft_consensus.cc:1671] finished
0806 03:43:31.974252 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:31.974675 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":941168,"threads_started":1,"wal-append.queue_time_us":1070093}
I20260806 03:43:32.100016 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:32.100944 665 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:31.976315 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2167) took 2164 ms. Trace:
I20260806 03:43:31.977700 668 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260806 03:43:31.707520 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 78436us
W20260806 03:43:31.978602 755 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.001s sys 0.013s
I20260806 03:43:31.646191 766 rpcz_store.cc:276] 0806 03:43:30.428058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.441765 (+ 13707us) service_pool.cc:227] Handling call
0806 03:43:30.544467 (+102702us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.632092 (+ 87625us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.638495 (+ 6403us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.716447 (+ 77952us) log.cc:842] Serialized 26 byte log entry
0806 03:43:31.185662 (+469215us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:31.254314 (+ 68652us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.283419 (+ 29105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:31.493033 (+209614us) raft_consensus.cc:1671] finished
0806 03:43:31.493073 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:31.493270 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7445,"raft.queue_time_us":5130,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":17608,"thread_start_us":304697,"threads_started":1,"wal-append.queue_time_us":774567}
W20260806 03:43:32.126693 755 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.714s user 0.001s sys 0.014s
W20260806 03:43:31.750497 671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.002s sys 0.010s
W20260806 03:43:32.127874 671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.097s user 0.002s sys 0.010s
I20260806 03:43:32.129675 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2190) took 1174 ms. Trace:
W20260806 03:43:31.778077 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 54248.3us
W20260806 03:43:31.983109 767 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s user 0.008s sys 0.008s
W20260806 03:43:32.130916 767 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s user 0.008s sys 0.008s
I20260806 03:43:31.983462 770 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
W20260806 03:43:31.984158 768 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.025s sys 0.041s
W20260806 03:43:32.132596 768 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.104s user 0.025s sys 0.041s
I20260806 03:43:31.796502 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2105) took 1081 ms. Trace:
W20260806 03:43:31.820950 773 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s user 0.008s sys 0.007s
I20260806 03:43:32.134687 3647 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:32.134246 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2165) took 2323 ms. Trace:
I20260806 03:43:32.135737 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1233) took 1582 ms. Trace:
W20260806 03:43:31.991884 775 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.000s sys 0.015s
I20260806 03:43:32.132627 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.136829 3647 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:32.134024 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 425) took 1518 ms. Trace:
I20260806 03:43:32.137461 3647 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:31.822835 1622 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.831s user 0.024s sys 0.048s
I20260806 03:43:32.134246 776 rpcz_store.cc:276] 0806 03:43:30.715043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.649314 (+934271us) service_pool.cc:227] Handling call
0806 03:43:31.649560 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:31.796230 (+146670us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.796443 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:31.836076 1665 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.750s user 0.013s sys 0.004s
W20260806 03:43:32.142915 1665 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.057s user 0.014s sys 0.005s
I20260806 03:43:31.831148 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:31.850863 1668 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:31.828722 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:32.012631 1658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.284s user 0.033s sys 0.053s
I20260806 03:43:31.810299 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 324) took 1273 ms. Trace:
W20260806 03:43:31.824781 1662 rpcz_store.cc:269] 0806 03:43:30.032201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.074282 (+ 42081us) service_pool.cc:227] Handling call
0806 03:43:31.700345 (+1626063us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:31.873262 3332 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.013s sys 0.025s
I20260806 03:43:31.870294 2911 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:32.195634 3527 consensus_queue.cc:1035] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260806 03:43:32.208698 3815 tablet_replica.cc:330] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: stopping tablet replica
I20260806 03:43:32.208513 2911 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:31.858660 3399 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:31.885787 3419 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:31.907312 3615 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s user 0.011s sys 0.006s
I20260806 03:43:32.211513 3604 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.213662 3419 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:31.924883 3725 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:31.915539 3770 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.927626 3698 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.217309 3698 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:31.917833 3651 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.089728 3618 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:32.109826 3621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s user 0.008s sys 0.013s
I20260806 03:43:32.220966 3651 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:32.137059 775 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.854s user 0.000s sys 0.015s
I20260806 03:43:32.223099 664 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:32.223857 669 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.016s sys 0.032s
W20260806 03:43:32.224287 669 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.016s sys 0.032s
I20260806 03:43:32.003749 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:32.225289 674 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.227548 3615 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:32.228246 3615 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.094136 563 rpcz_store.cc:276] 0806 03:43:30.824094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.009019 (+184925us) service_pool.cc:227] Handling call
0806 03:43:32.093210 (+1084191us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b54001c7fd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:32.093452 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2025456896}
W20260806 03:43:32.008728 1624 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.925s user 0.000s sys 0.016s
W20260806 03:43:32.231415 1624 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.148s user 0.000s sys 0.016s
W20260806 03:43:32.232007 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.14895e+06us
I20260806 03:43:32.137557 770 rpcz_store.cc:276] 0806 03:43:30.615227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.075480 (+460253us) service_pool.cc:227] Handling call
0806 03:43:31.075790 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:32.132462 (+1056672us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.133542 (+ 1080us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b54000fffd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:32.133897 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2085536000}
I20260806 03:43:32.136247 773 rpcz_store.cc:276] 0806 03:43:30.553246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.066045 (+512799us) service_pool.cc:227] Handling call
0806 03:43:31.069499 (+ 3454us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:31.069806 (+ 307us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:31.069845 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:31.070624 (+ 779us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.135092 (+1064468us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:32.135183 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.135270 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.135338 (+ 68us) raft_consensus.cc:1671] finished
0806 03:43:32.135348 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.135556 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10112,"thread_start_us":1063805,"threads_started":1,"wal-append.queue_time_us":841116}
I20260806 03:43:32.234288 1660 rpcz_store.cc:276] 0806 03:43:30.536691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.758855 (+222164us) service_pool.cc:227] Handling call
0806 03:43:31.776583 (+1017728us) spinlock_profiling.cc:243] Waited 664 ms on lock 0x7b54004f5e54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:31.795563 (+ 18980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1526685056}
I20260806 03:43:32.130218 671 rpcz_store.cc:276] 0806 03:43:30.955208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.955657 (+ 449us) service_pool.cc:227] Handling call
0806 03:43:30.960441 (+ 4784us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.960849 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.960887 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.979459 (+ 18572us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.128754 (+1149295us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:32.128871 (+ 117us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.129222 (+ 351us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.129304 (+ 82us) raft_consensus.cc:1671] finished
0806 03:43:32.129321 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.129577 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10,"raft.queue_time_us":536,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":22493,"thread_start_us":1097895,"threads_started":1,"wal-append.queue_time_us":883959}
I20260806 03:43:32.102600 667 rpcz_store.cc:276] 0806 03:43:29.811474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.649977 (+838503us) service_pool.cc:227] Handling call
0806 03:43:30.650549 (+ 572us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.655500 (+ 4951us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.655532 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.697516 (+ 41984us) log.cc:842] Serialized 26 byte log entry
0806 03:43:31.714677 (+1017161us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:31.714735 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:31.714837 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:31.975750 (+260913us) raft_consensus.cc:1671] finished
0806 03:43:31.975799 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:31.976115 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10994,"raft.queue_time_us":6483,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":11186,"thread_start_us":987311,"threads_started":1,"wal-append.queue_time_us":1139784}
I20260806 03:43:32.135560 663 rpcz_store.cc:276] 0806 03:43:29.810768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.636165 (+825397us) service_pool.cc:227] Handling call
0806 03:43:30.675652 (+ 39487us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.675918 (+ 266us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:31.014138 (+338220us) log.cc:842] Serialized 14 byte log entry
0806 03:43:32.133558 (+1119420us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:32.133669 (+ 111us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:32.133789 (+ 120us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.133924 (+ 135us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.134160 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41813,"prepare.queue_time_us":814721,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":40844,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":205,"thread_start_us":1376609,"threads_started":2,"wal-append.queue_time_us":894495}
I20260806 03:43:32.252310 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2091) took 1824 ms. Trace:
I20260806 03:43:32.252888 768 rpcz_store.cc:276] 0806 03:43:30.428343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.991007 (+562664us) service_pool.cc:227] Handling call
0806 03:43:30.991404 (+ 397us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:30.991750 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:30.991774 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:30.992272 (+ 498us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.133458 (+1141186us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:32.133592 (+ 134us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.133729 (+ 137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.251939 (+118210us) raft_consensus.cc:1671] finished
0806 03:43:32.251960 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.252229 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16525,"raft.queue_time_us":2313,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":14848,"thread_start_us":1104067,"threads_started":1,"wal-append.queue_time_us":1246586}
I20260806 03:43:32.187865 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2189) took 1277 ms. Trace:
I20260806 03:43:32.175865 501 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:32.200171 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.19211e+06us
I20260806 03:43:32.210616 771 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:31.920737 3694 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.122840 3586 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:32.271085 3586 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:32.273162 3586 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.221092 3618 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:32.275233 3586 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.264598 2911 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.276393 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:32.277978 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 133408us
I20260806 03:43:32.280350 3814 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:32.289700 3518 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:32.290585 3518 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.292369 3332 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:32.293144 3332 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.295128 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:32.298142 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61400.1us
I20260806 03:43:32.298909 3770 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:32.301219 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.258102 668 rpcz_store.cc:276] 0806 03:43:30.910294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.910513 (+ 219us) service_pool.cc:227] Handling call
0806 03:43:30.935252 (+ 24739us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:32.148182 (+1212930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.172387 (+ 24205us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002cdb54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:32.187770 (+ 15383us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2395291904}
I20260806 03:43:32.302354 768 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:32.265522 3583 raft_consensus.cc:2741] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260806 03:43:32.236873 1658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.519s user 0.034s sys 0.054s
I20260806 03:43:32.306576 3814 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:32.307222 3604 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.307430 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 667) took 1877 ms. Trace:
I20260806 03:43:32.307731 3616 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:32.308465 3616 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.311092 3616 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.307976 1658 rpcz_store.cc:276] 0806 03:43:30.429632 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.429769 (+ 137us) service_pool.cc:227] Handling call
0806 03:43:30.430120 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.430534 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:43:30.698542 (+268008us) log.cc:842] Serialized 14 byte log entry
0806 03:43:32.306785 (+1608243us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:32.306903 (+ 118us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:32.307047 (+ 144us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.307142 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.307337 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":290891,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":1707,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":150,"thread_start_us":1819085,"threads_started":2,"wal-append.queue_time_us":1539502}
I20260806 03:43:32.311913 3518 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.302284 3651 raft_consensus.cc:2741] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260806 03:43:32.264582 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 132415us
I20260806 03:43:32.314848 3332 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:32.317873 3609 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.840s user 0.010s sys 0.007s
I20260806 03:43:32.318986 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.319814 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.325229 771 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.327574 728 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.329677 3583 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:32.321290 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 420) took 1866 ms. Trace:
I20260806 03:43:32.347504 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.348835 665 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:32.345935 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.338002 767 rpcz_store.cc:276] 0806 03:43:30.455218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.043225 (+588007us) service_pool.cc:227] Handling call
0806 03:43:31.043787 (+ 562us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:31.188610 (+144823us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:31.188643 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:31.189113 (+ 470us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.131769 (+942656us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:32.131898 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.132066 (+ 168us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.237340 (+105274us) raft_consensus.cc:1671] finished
0806 03:43:32.237370 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.321209 (+ 83839us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143910,"raft.queue_time_us":192621,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":32118,"thread_start_us":942002,"threads_started":1,"wal-append.queue_time_us":1044331}
I20260806 03:43:32.366632 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.367835 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.373104 563 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:32.351311 3332 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:32.376515 773 raft_consensus.cc:2396] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 03:43:32.365790 3328 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:32.390599 3450 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.009s
I20260806 03:43:32.390843 616 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:32.391183 3450 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.392361 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.392825 3770 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:32.393298 3770 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:32.393564 3450 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.393679 3770 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:32.395743 3365 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.578s user 0.010s sys 0.010s
I20260806 03:43:32.396003 3399 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.396260 3365 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.402618 3604 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:32.403255 3604 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:32.403748 3604 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.404285 770 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:32.391618 3328 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:32.387877 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.002s sys 0.010s
W20260806 03:43:32.415545 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.044s user 0.003s sys 0.010s
W20260806 03:43:32.416055 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 1.04445e+06us
I20260806 03:43:32.416352 772 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:32.383891 3615 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.418459 3234 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:32.388446 661 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.042s sys 0.054s
I20260806 03:43:32.411855 3647 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:32.413149 614 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.548s (SENT) [suppressed 44 similar messages]
I20260806 03:43:32.350996 3609 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.419849 674 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
W20260806 03:43:32.428934 661 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s user 0.043s sys 0.054s
I20260806 03:43:32.431949 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.431267 771 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:32.439495 773 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:43:32.438076 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1224) took 1987 ms. Trace:
I20260806 03:43:32.437893 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:32.442473 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 54035.8us
I20260806 03:43:32.449932 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:32.449298 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.389571 3332 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.453132 776 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:32.446787 499 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:32.455332 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 107967us
I20260806 03:43:32.458039 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2171) took 2644 ms. Trace:
I20260806 03:43:32.454846 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:32.435487 3365 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.443233 775 rpcz_store.cc:276] 0806 03:43:30.450576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.007630 (+557054us) service_pool.cc:227] Handling call
0806 03:43:31.187095 (+179465us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:31.187308 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:43:32.221924 (+1034616us) log.cc:842] Serialized 14 byte log entry
0806 03:43:32.426211 (+204287us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:32.432735 (+ 6524us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:32.435333 (+ 2598us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.437772 (+ 2439us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.437999 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2870,"prepare.queue_time_us":665467,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":1353,"raft.queue_time_us":41090,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":14088,"thread_start_us":1090184,"threads_started":2}
I20260806 03:43:32.486902 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:32.487402 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.488032 1662 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:32.488372 3616 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:32.488729 3499 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:32.488965 3422 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.783s user 0.010s sys 0.010s
I20260806 03:43:32.489578 3596 consensus_queue.cc:1046] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 03:43:32.489434 3499 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:32.490135 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 131781us
I20260806 03:43:32.490533 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.491181 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.492036 1664 raft_consensus.cc:2354] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:32.490743 3596 raft_consensus.cc:3032] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:32.488023 499 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.463356 3609 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.496996 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.458647 661 rpcz_store.cc:276] 0806 03:43:29.813298 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:30.651031 (+837733us) service_pool.cc:227] Handling call
0806 03:43:30.651317 (+ 286us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:30.651531 (+ 214us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:31.418824 (+767293us) log.cc:842] Serialized 14 byte log entry
0806 03:43:32.449609 (+1030785us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:32.457539 (+ 7930us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:32.457638 (+ 99us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.457762 (+ 124us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.457964 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":838815,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":95,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":12539,"thread_start_us":1748507,"threads_started":2,"wal-append.queue_time_us":828792}
I20260806 03:43:32.497892 768 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:32.488463 1663 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:32.492219 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.498708 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:32.501791 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:32.502902 3527 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:32.503351 3647 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.503552 3527 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.489452 3422 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.447724 3618 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.503027 563 raft_consensus.cc:2396] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:32.491904 3432 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:32.519322 772 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:32.514436 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.518814 3710 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:32.495154 3596 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:32.530745 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 51182.6us
I20260806 03:43:32.506163 3422 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:32.528911 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.535312 663 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.535529 3710 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.556700 3450 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:32.557132 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:32.558177 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:32.558073 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.529203 3586 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:32.506166 3527 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:32.527915 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.599110 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.605990 770 raft_consensus.cc:2379] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:32.619765 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 201837us
W20260806 03:43:32.620477 1622 connection.cc:232] RPC call timeout handler was delayed by -1.4410152910277247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
W20260806 03:43:32.620954 1622 connection.cc:232] RPC call timeout handler was delayed by -1.4032152909785509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2235s for a response
W20260806 03:43:32.621316 1622 connection.cc:232] RPC call timeout handler was delayed by -1.0819152910262346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25920000000000004s for a response
I20260806 03:43:32.634716 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:32.568706 3450 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:32.594060 3814 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:32.562783 3675 consensus_queue.cc:1046] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260806 03:43:32.593812 3586 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.644841 772 raft_consensus.cc:1261] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:32.592830 3621 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260806 03:43:32.679500 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.644423 3450 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:32.719605 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 88761.4us
I20260806 03:43:32.674274 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:32.754421 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 266612us
I20260806 03:43:32.776201 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:32.779578 3618 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:32.781368 3710 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:32.692973 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.782862 3332 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:32.783643 3586 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.768985 3596 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.785282 3596 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.785187 499 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.697674 3604 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.789263 3604 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.790901 775 raft_consensus.cc:2379] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:32.792861 3834 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
I20260806 03:43:32.790961 3770 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.670655 3814 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:32.801237 3814 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:32.720372 3831 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260806 03:43:32.720137 3357 consensus_meta.cc:287] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.685s user 0.011s sys 0.007s
I20260806 03:43:32.804976 3357 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.720001 2911 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:32.720757 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 671) took 2287 ms. Trace:
I20260806 03:43:32.807595 1665 rpcz_store.cc:276] 0806 03:43:30.433439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.085018 (+651579us) service_pool.cc:227] Handling call
0806 03:43:31.085377 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:32.161754 (+1076377us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:32.173196 (+ 11442us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:32.178037 (+ 4841us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.367467 (+189430us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:32.595817 (+228350us) log.cc:842] Serialized 14 byte log entry
0806 03:43:32.644307 (+ 48490us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.675191 (+ 30884us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.679239 (+ 4048us) raft_consensus.cc:1671] finished
0806 03:43:32.699634 (+ 20395us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.720674 (+ 21040us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101880,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":7390,"thread_start_us":1370437,"threads_started":3,"wal-append.queue_time_us":192335}
I20260806 03:43:32.803797 3770 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:32.812019 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1244) took 1187 ms. Trace:
I20260806 03:43:32.812561 769 rpcz_store.cc:276] 0806 03:43:31.623992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.800477 (+176485us) service_pool.cc:227] Handling call
0806 03:43:31.800900 (+ 423us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:32.330625 (+529725us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:32.330676 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:32.331263 (+ 587us) log.cc:842] Serialized 26 byte log entry
0806 03:43:32.463241 (+131978us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:32.463290 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:32.474663 (+ 11373us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:32.811662 (+336999us) raft_consensus.cc:1671] finished
0806 03:43:32.811683 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:32.811820 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1377,"raft.queue_time_us":26575,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4184,"thread_start_us":516420,"threads_started":2,"wal-append.queue_time_us":262204}
W20260806 03:43:32.814677 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 322615us
I20260806 03:43:32.816167 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.817570 615 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:32.819136 3621 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:32.818800 3450 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.819454 3815 raft_consensus.cc:2229] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:43:32.819577 3621 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:32.820050 772 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:32.820958 3815 raft_consensus.cc:2258] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:43:32.820757 3621 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.824471 3815 ts_tablet_manager.cc:1905] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260806 03:43:32.825402 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 203113us
I20260806 03:43:32.826664 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:32.828398 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.830238 3614 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:32.830799 3614 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:32.831251 3614 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:32.834903 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 65870.5us
I20260806 03:43:32.834671 3651 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:32.835551 3651 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:32.835721 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:32.835980 3651 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:32.838044 3599 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:32.838630 3599 raft_consensus.cc:686] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:32.839166 2911 raft_consensus.cc:686] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:32.840266 3621 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:32.840288 2911 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.843982 3710 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:32.844266 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 59208.1us
I20260806 03:43:32.839752 3599 consensus_queue.cc:237] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:32.845134 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.846012 3710 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:32.846441 3710 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:32.846809 3710 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.847137 771 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:32.835268 3403 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:32.847769 3403 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:32.848170 3403 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:32.851351 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.852176 771 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:32.852953 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:32.860015 499 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.862664 674 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:32.865756 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:32.866797 674 raft_consensus.cc:2379] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 2: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:32.868114 3675 consensus_queue.cc:1046] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:32.869267 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:32.870362 773 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:32.871241 3703 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:32.871743 3703 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:32.872153 3703 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:32.872891 1622 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:32.877557 3357 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:32.879678 499 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.886845 3357 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:32.887872 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:32.896019 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 358791us
I20260806 03:43:32.920071 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:32.921029 3604 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:32.901929 499 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:32.921483 3604 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:32.922088 3604 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.922726 499 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:32.923609 3615 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:32.923723 3614 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.924304 3615 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:32.924795 3615 raft_consensus.cc:686] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:32.925968 3614 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:32.926097 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.926924 562 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:32.925799 3615 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.928089 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.893905 3651 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.897333 3703 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:32.637441 1657 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:32.903314 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.904546 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:32.904803 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:32.906661 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:32.911655 614 leader_election.cc:334] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.993s (SENT)
I20260806 03:43:32.913255 659 raft_consensus.cc:1226] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:32.991734 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 118936us
W20260806 03:43:32.995536 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 72938.3us
I20260806 03:43:32.917573 3621 raft_consensus.cc:3032] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:33.003286 562 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
W20260806 03:43:32.934904 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 117416us
I20260806 03:43:32.940145 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.035988 1664 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:32.985240 663 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:33.038540 3770 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:32.934679 3332 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:32.948530 3403 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.075205 548 raft_consensus.cc:1226] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:33.075897 3403 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:33.061872 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:33.035162 3621 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:33.066262 3586 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:33.066109 1657 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 684) took 1684 ms (client timeout 1888 ms). Trace:
W20260806 03:43:33.077716 1657 rpcz_store.cc:269] 0806 03:43:31.379451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.797262 (+417811us) service_pool.cc:227] Handling call
0806 03:43:33.054397 (+1257135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:33.068153 3651 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:33.053238 3647 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:33.079221 3599 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:33.079809 3599 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:33.080403 3599 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:33.081634 1658 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:33.082576 3432 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:33.083446 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.062232 2911 consensus_queue.cc:1046] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:33.084396 3703 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.084724 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.085508 558 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:33.088119 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.089359 1658 raft_consensus.cc:2362] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4. Re-sending same reply.
I20260806 03:43:33.090489 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.091277 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.091992 561 raft_consensus.cc:2362] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4. Re-sending same reply.
I20260806 03:43:33.092284 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.093618 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.093918 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.095271 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.096112 558 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:33.096380 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.096922 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.097339 1663 raft_consensus.cc:2379] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:32.965036 3613 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.076686 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.078805 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.081019 3621 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.069384 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.088294 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.084801 3596 raft_consensus.cc:3032] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:33.069397 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.097661 547 raft_consensus.cc:2354] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4.
I20260806 03:43:32.956910 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.092856 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:32.949518 3450 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.132699 3803 raft_consensus.cc:3032] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
W20260806 03:43:33.135097 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 195160us
I20260806 03:43:33.098839 558 raft_consensus.cc:2354] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8.
I20260806 03:43:33.103449 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.126601 562 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:33.127382 3596 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:33.128645 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.129410 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.129623 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:33.130560 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61302.8us
I20260806 03:43:33.131106 3651 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:33.131517 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.160291 563 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:33.160823 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:33.161170 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.161334 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.161971 562 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:32.948347 3710 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.162268 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:33.163040 770 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:33.163914 3710 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.164558 3596 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.164575 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.171128 1662 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:33.159986 1671 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:33.161368 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:32.974997 3604 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.187865 3604 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:33.155068 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.189520 674 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:43:33.140407 3621 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:33.192613 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.192986 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.193427 661 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:33.135464 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.194628 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:33.071504 3770 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:33.194960 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.195221 661 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:33.195390 3770 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:33.138149 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 226550us
I20260806 03:43:33.164232 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.196149 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.099717 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.180099 770 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:33.180751 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.180821 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.154680 3803 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:43:33.158352 3234 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.738s user 0.011s sys 0.002s
I20260806 03:43:33.175544 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.183414 3615 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.184512 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.185231 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.185621 615 leader_election.cc:398] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:33.219417 615 leader_election.cc:401] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:43:33.166144 3596 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.137935 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.191267 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.193749 665 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:33.104204 1663 raft_consensus.cc:1226] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:33.177636 562 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.303356 1660 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:33.303546 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.304210 562 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:33.305125 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.305747 562 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.306118 563 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.306529 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.136786 766 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:33.133834 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 267223us
I20260806 03:43:33.307899 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.309304 3613 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:33.205528 544 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.310202 3862 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:33.311391 3613 consensus_queue.cc:1046] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:33.313158 544 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:33.313480 3613 raft_consensus.cc:3032] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:33.313951 3613 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:33.207641 1668 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:33.101524 3814 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.213146 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.317255 558 raft_consensus.cc:1261] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 03:43:33.225220 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.216987 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:33.199759 3599 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.223053 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:33.218520 674 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:33.222736 3615 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.322470 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:33.323139 768 raft_consensus.cc:2379] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:33.240878 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:33.244870 3815 ts_tablet_manager.cc:1918] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 03:43:33.250929 547 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:33.251322 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.252229 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.252959 1658 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:33.240752 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.248441 3770 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.328171 3815 log.cc:1196] T 20fcbc41710a42cd9ee5dacc5d9577fb P 0374a84d5e9f4e3c8d8287e2382379c8: Deleting WAL directory at /tmp/dist-test-taskRSO0pM/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785987679226243-31273-0/minicluster-data/ts-2-root/wals/20fcbc41710a42cd9ee5dacc5d9577fb
I20260806 03:43:33.206351 3651 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:33.240447 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 76359.7us
I20260806 03:43:33.238955 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:33.239989 1665 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 03:43:33.263387 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 77992.8us
I20260806 03:43:33.264094 3647 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.265641 3499 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.271261 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:33.272672 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.274106 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:33.274466 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:33.274557 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:33.343425 770 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:33.275445 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.344339 659 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:33.278788 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 54919.6us
I20260806 03:43:33.207150 548 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.281454 1664 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:33.285892 767 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:33.343017 771 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:33.348356 728 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:33.241583 3803 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.301007 663 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:43:33.318643 558 raft_consensus.cc:1226] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:33.214954 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:33.319307 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:33.324585 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 83784.2us
I20260806 03:43:33.333040 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:33.353092 768 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:33.332723 547 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:33.223284 3432 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.332396 3334 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:43:33.343044 773 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.346819 544 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:43:33.328348 3703 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:33.352979 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:33.336115 562 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:33.349187 3604 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:43:33.368821 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 96304.8us
I20260806 03:43:33.360105 3703 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:33.373204 546 raft_consensus.cc:1261] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:33.374490 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.365054 564 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:33.378129 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.379043 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.379725 499 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.380203 3710 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:33.380669 3710 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:33.381081 3710 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:33.381790 3815 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47796 (request call id 921) took 1972 ms. Trace:
I20260806 03:43:33.359901 3499 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:33.382800 3499 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:33.382212 3815 rpcz_store.cc:276] 0806 03:43:31.409093 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:31.411025 (+ 1932us) service_pool.cc:227] Handling call
0806 03:43:32.207374 (+796349us) ts_tablet_manager.cc:1145] Deleting tablet 20fcbc41710a42cd9ee5dacc5d9577fb
0806 03:43:32.207554 (+ 180us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0806 03:43:32.988536 (+780982us) tablet_metadata.cc:641] Metadata flushed
0806 03:43:33.244614 (+256078us) tablet_metadata.cc:641] Metadata flushed
0806 03:43:33.381658 (+137044us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":609021,"tablet-delete.queue_time_us":795185,"thread_start_us":714976,"threads_started":1}
I20260806 03:43:33.383900 3499 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.385601 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.386029 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.386437 3357 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.378228 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.387696 3586 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.388204 3586 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.379060 1664 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:33.389701 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.390558 3596 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:33.391429 3803 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:33.391692 3770 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.393606 3703 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:33.361919 3234 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 03:43:33.395272 3618 consensus_queue.cc:1046] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:33.396191 3234 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:33.397367 3234 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:33.397917 3234 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:33.398671 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 179385us
I20260806 03:43:33.399480 501 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:33.399971 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 147866us
I20260806 03:43:33.403313 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.414265 1664 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:33.373965 3604 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:33.419339 3604 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:33.419277 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 58877.1us
I20260806 03:43:33.420449 1622 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:33.402007 3586 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.425329 3651 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:33.376591 3613 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.427410 3615 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:33.427534 3613 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.428242 3596 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:33.428491 3814 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:33.429790 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 55498.3us
I20260806 03:43:33.430037 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.430783 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.357003 3432 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.418609 3770 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:33.418799 3599 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.400003 3450 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:33.426076 3651 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:33.437022 501 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.440840 3710 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.441867 3618 raft_consensus.cc:3032] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:33.442606 3599 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:33.457135 3618 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:33.447347 3647 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:33.448465 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.398339 3234 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.458189 3609 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:33.442019 3432 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.456171 3770 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:33.456535 548 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.456673 614 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:33.450201 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.471087 727 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.434906 3357 raft_consensus.cc:3032] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:33.464639 659 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260806 03:43:33.466969 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 44718.3us
I20260806 03:43:33.435153 3862 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:33.467922 3332 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:33.471833 3357 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:33.464493 1658 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:33.458069 3618 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.463157 3599 raft_consensus.cc:686] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:33.465674 3586 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:43:33.509807 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 58100.6us
I20260806 03:43:33.468530 3814 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:33.465983 3450 raft_consensus.cc:686] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:33.462792 3621 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 03:43:33.487824 3647 consensus_queue.cc:1046] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260806 03:43:33.487434 3862 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:33.566938 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.567866 1664 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:33.493757 3332 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:33.514793 3599 consensus_queue.cc:237] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.589210 3586 raft_consensus.cc:2741] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 03:43:33.587054 3334 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:33.590553 3334 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.591015 3604 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:33.515362 3770 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.593115 3604 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:33.592872 3450 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.595432 1664 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:33.513022 3609 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.600152 615 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:33.603389 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 133058us
W20260806 03:43:33.606042 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 97325.6us
I20260806 03:43:33.583961 3862 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.609905 501 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.526060 402 catalog_manager.cc:5495] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 3, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:33.595865 3618 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:33.610775 3710 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:33.613246 402 catalog_manager.cc:5495] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:33.605146 3518 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260806 03:43:33.628707 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:33.627570 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 237910us
W20260806 03:43:33.632958 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 284680us
I20260806 03:43:33.634248 402 catalog_manager.cc:5495] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:33.619653 3814 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.605046 3609 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:33.636101 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 269423us
I20260806 03:43:33.596001 3651 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.641865 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:33.645450 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 171993us
I20260806 03:43:33.645843 402 catalog_manager.cc:5495] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 4, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:33.651103 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:33.632449 3703 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.674883 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:33.676623 402 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 4 to 6, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 6 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:33.673240 663 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:33.627976 3332 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.679526 378 catalog_manager.cc:4841] TS 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937): tablet 20fcbc41710a42cd9ee5dacc5d9577fb (table test-workload [id=0a70905edfcd49088d3d66a127c1a9e6]) successfully deleted
I20260806 03:43:33.680172 3332 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.614166 3357 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.683656 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.684478 3357 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:33.684878 3887 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:33.685899 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.687176 674 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:33.690487 3887 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:43:33.692318 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 58358.7us
I20260806 03:43:33.700539 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:33.685978 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 65030.5us
I20260806 03:43:33.703855 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:33.706291 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 144201us
I20260806 03:43:33.706535 402 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:43:33.708442 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 66671.2us
I20260806 03:43:33.670576 3621 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:33.703169 773 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:33.708603 402 catalog_manager.cc:5495] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:33.720340 498 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:33.671984 3234 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:33.691013 3609 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:33.690970 3887 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:33.722577 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 210328us
I20260806 03:43:33.729521 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.729753 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:33.730633 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.731344 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:33.738327 1671 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:33.738477 1670 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:33.705214 3861 raft_consensus.cc:3032] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:33.691399 3814 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.734232 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:33.733613 3647 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:33.717288 3432 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.719894 3621 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:33.749971 3814 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.752295 3814 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:33.752338 548 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20df53c445204494999245a03bc56a91"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:33.753161 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:33.753896 663 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:33.754739 3892 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:33.755175 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:33.756156 3814 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:33.756588 3814 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:33.756976 3814 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:33.753224 548 raft_consensus.cc:595] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.766458 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.767730 501 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.706843 3615 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260806 03:43:33.737094 3710 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:33.762535 3621 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.706543 3596 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260806 03:43:33.739982 3861 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:33.737550 3887 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:33.775465 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.727305 3613 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 03:43:33.778504 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:33.727567 3499 consensus_queue.cc:1046] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260806 03:43:33.766276 3888 raft_consensus.cc:3032] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:33.791821 2911 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:33.771906 3647 raft_consensus.cc:686] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:33.773154 3450 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:33.775591 3710 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:33.802997 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 74401.1us
W20260806 03:43:33.803215 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 69086.6us
I20260806 03:43:33.795229 3888 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:33.782533 3518 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
W20260806 03:43:33.823246 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 80785.5us
I20260806 03:43:33.820770 3874 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.798190 3432 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 03:43:33.827399 3647 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:33.828857 3877 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260806 03:43:33.829113 3770 raft_consensus.cc:3032] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:33.829531 3770 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:33.830681 3703 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:33.831219 3882 log.cc:924] Time spent T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260806 03:43:33.830729 3880 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:33.833768 3861 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.835204 3861 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:33.806584 3710 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:33.854511 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46492.9us
I20260806 03:43:33.849476 615 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:33.867570 3888 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.868893 3888 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:33.888729 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 50138us
W20260806 03:43:33.893011 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 53614.2us
W20260806 03:43:33.904321 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 129582us
I20260806 03:43:33.905900 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:33.802567 3450 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:33.916649 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:33.918200 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 91921.3us
I20260806 03:43:33.898967 3892 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:33.850420 3770 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:33.971045 499 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:34.164163 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:34.200362 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 229409us
I20260806 03:43:33.869357 3902 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:33.919246 3332 raft_consensus.cc:3032] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:43:33.985324 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 217028us
I20260806 03:43:34.214030 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:34.214758 767 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:34.204633 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 284186us
I20260806 03:43:34.088419 2911 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:34.002974 614 leader_election.cc:334] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.423s (SENT)
I20260806 03:43:34.200452 1668 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:34.087473 1671 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:34.210131 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:34.190732 1660 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.887s user 0.012s sys 0.001s
I20260806 03:43:34.221660 773 raft_consensus.cc:1261] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:34.203263 3874 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:34.213079 3892 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:34.035205 3609 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:34.229933 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:34.233369 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:34.243624 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:34.252512 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:34.256896 767 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:34.229628 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:34.226631 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:34.254262 771 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:34.229269 766 raft_consensus.cc:1261] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:43:34.247193 776 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:34.087162 3803 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:34.062466 3905 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:43:34.141933 1662 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 03:43:34.228916 3898 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20260806 03:43:34.236989 3912 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
I20260806 03:43:34.262917 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:34.213245 3332 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:34.235944 3903 log.cc:924] Time spent T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
W20260806 03:43:34.314344 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 58269.7us
I20260806 03:43:34.269690 1671 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:43:34.315428 1668 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
W20260806 03:43:34.333479 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 355055us
I20260806 03:43:34.222612 1660 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:34.254588 3334 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:34.234928 3770 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:34.258960 3874 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:34.369670 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:34.321887 3876 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:34.319051 3450 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:34.388773 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 472202us
I20260806 03:43:34.389492 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:34.391357 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:34.388334 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 66343.2us
I20260806 03:43:34.447042 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:34.378749 3862 consensus_meta.cc:287] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.756s user 0.011s sys 0.001s
I20260806 03:43:34.449061 1671 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:34.456468 728 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:34.457458 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:34.467438 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 77530.1us
I20260806 03:43:34.234592 3892 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:34.383842 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 698) took 1209 ms. Trace:
W20260806 03:43:34.451320 3814 consensus_meta.cc:287] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.694s user 0.010s sys 0.002s
I20260806 03:43:34.473263 659 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:43:34.450716 3651 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:43:34.350282 3803 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:34.451098 3450 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:34.382776 3334 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:34.498126 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 264812us
I20260806 03:43:34.502815 1670 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20260806 03:43:34.503192 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 500287us
I20260806 03:43:34.503242 1670 pending_rounds.cc:76] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting all ops after (but not including) 6
I20260806 03:43:34.495199 3921 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:34.504202 1670 pending_rounds.cc:97] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 03:43:34.510921 1670 raft_consensus.cc:2866] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:34.488858 3651 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:34.515033 1622 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:34.527181 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 137765us
I20260806 03:43:34.479861 3916 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:34.459424 3862 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:34.469904 3874 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:34.532661 616 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:34.479615 1660 rpcz_store.cc:276] 0806 03:43:33.174490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:33.190958 (+ 16468us) service_pool.cc:227] Handling call
0806 03:43:34.379672 (+1188714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:34.502414 3876 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:34.545010 616 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:34.490262 3814 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:34.527139 3604 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:34.561923 3878 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 03:43:34.500387 3450 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:34.532706 3651 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:34.490486 3332 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:34.600500 3614 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:34.620348 3876 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:34.621336 3924 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260806 03:43:34.532567 3234 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260806 03:43:34.618592 3604 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:34.623930 3604 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:34.624264 3604 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:34.625023 3621 consensus_meta.cc:287] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.835s user 0.009s sys 0.003s
I20260806 03:43:34.625488 3621 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:34.626958 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 170620us
I20260806 03:43:34.541883 3922 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:43:34.633872 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 186911us
I20260806 03:43:34.646911 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:34.620379 3914 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20260806 03:43:34.506212 3921 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:34.647816 775 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:34.629747 1665 raft_consensus.cc:1226] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:34.662539 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 169211us
W20260806 03:43:34.649181 3888 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.780s user 0.012s sys 0.004s
I20260806 03:43:34.641446 3647 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:34.657073 3922 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:34.643558 3905 raft_consensus.cc:686] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:34.666112 1658 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:34.713850 616 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:34.716118 775 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:43:34.696668 3332 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:34.722604 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 221027us
I20260806 03:43:34.698647 3647 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:34.699144 3917 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:34.733261 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 147469us
I20260806 03:43:34.744602 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:34.745321 1622 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:34.714720 3883 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:34.741923 3647 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:34.767584 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:34.768658 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:34.771564 3940 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:34.772094 3940 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:34.772298 3862 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:34.772459 3940 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:34.772702 3874 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:34.739212 3922 raft_consensus.cc:686] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:34.773903 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:34.774755 546 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:34.775679 3450 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:34.777097 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:34.747510 3803 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:34.769323 564 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:34.793064 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:34.776341 3450 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:43:34.793773 775 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:34.756582 3878 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
I20260806 03:43:34.780894 3876 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:34.748943 3917 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:34.773990 3922 consensus_queue.cc:237] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.441314 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:35.088943 767 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:34.822955 659 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:35.390621 3942 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.445124 3942 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.110666 3961 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:35.446976 3961 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:35.004985 3934 log.cc:924] Time spent T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.137s user 0.000s sys 0.001s
W20260806 03:43:34.991735 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 276045us
I20260806 03:43:35.449152 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:35.449888 771 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:35.449981 3961 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:35.127859 3921 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:35.062321 3964 log.cc:924] Time spent T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 03:43:35.116662 3960 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.451215 616 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:34.827195 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:35.471783 542 raft_consensus.cc:2362] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5. Re-sending same reply.
W20260806 03:43:34.896867 3357 consensus_meta.cc:287] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.212s user 0.012s sys 0.003s
W20260806 03:43:35.432940 3968 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.332s user 0.000s sys 0.000s
W20260806 03:43:35.438625 3770 consensus_meta.cc:287] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.052s user 0.009s sys 0.005s
W20260806 03:43:35.023806 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 358577us
I20260806 03:43:34.777299 3334 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:35.192972 3887 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.397s user 0.013s sys 0.003s
I20260806 03:43:35.209177 3862 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.011636 3955 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.177402 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:35.453568 3883 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.000s sys 0.010s
W20260806 03:43:35.229032 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 459066us
W20260806 03:43:35.063064 3928 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260806 03:43:34.868986 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:34.850795 3621 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:34.827668 3905 consensus_queue.cc:237] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:34.817137 3814 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:35.456367 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:35.498441 3518 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.457981 3876 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.551214 3814 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:35.551412 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 682501us
I20260806 03:43:35.552150 501 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:35.553517 3876 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:34.906752 3933 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:35.387874 3948 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:35.554715 501 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:35.555764 3814 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:34.816290 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:35.440119 3965 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20260806 03:43:34.954648 3861 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.119s user 0.005s sys 0.010s
W20260806 03:43:34.831868 3609 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.581s user 0.010s sys 0.003s
I20260806 03:43:35.557448 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 709) took 1346 ms. Trace:
I20260806 03:43:35.557680 3609 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:35.158596 3978 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:35.558717 3978 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:35.482316 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:35.562155 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:35.482481 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.564988 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:35.483629 3890 log.cc:924] Time spent T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 1.417s user 0.003s sys 0.021s
I20260806 03:43:35.484463 659 raft_consensus.cc:1261] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:35.568596 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.488173 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 351) took 1281 ms. Trace:
I20260806 03:43:35.474962 3942 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:35.490079 3952 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
I20260806 03:43:35.490840 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.477118 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:35.558008 1663 rpcz_store.cc:276] 0806 03:43:34.211159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.212225 (+ 1066us) service_pool.cc:227] Handling call
0806 03:43:34.229317 (+ 17092us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:34.485508 (+256191us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:34.488379 (+ 2871us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:34.550206 (+ 61827us) log.cc:842] Serialized 26 byte log entry
0806 03:43:34.758886 (+208680us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:34.771136 (+ 12250us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:34.782740 (+ 11604us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:35.557115 (+774375us) raft_consensus.cc:1671] finished
0806 03:43:35.557164 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:35.557354 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10315,"raft.queue_time_us":141210,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5575,"thread_start_us":320743,"threads_started":2,"wal-append.queue_time_us":522784}
I20260806 03:43:35.573413 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:35.574443 668 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:35.575430 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.576810 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.062680 3944 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.570101 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:35.577895 3944 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.579279 3978 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:35.570931 1670 rpcz_store.cc:276] 0806 03:43:34.206434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.206565 (+ 131us) service_pool.cc:227] Handling call
0806 03:43:34.206892 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:34.511467 (+304575us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:35.487919 (+976452us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54004506d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:35.488104 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":679542528}
I20260806 03:43:35.580956 3944 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:34.910605 3906 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.581956 3906 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.584895 3906 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:35.555164 3948 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.117179 3937 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:35.585757 3937 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.157399 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:35.516985 3946 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.444s user 0.000s sys 0.001s
I20260806 03:43:35.520001 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 352) took 1312 ms. Trace:
I20260806 03:43:35.520221 3613 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:35.526546 3977 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:34.965680 3874 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.552s
I20260806 03:43:35.529037 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:35.456892 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.498435 3961 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:35.507990 3332 consensus_meta.cc:287] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.775s user 0.005s sys 0.007s
I20260806 03:43:35.499102 3922 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:43:35.509593 3883 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.739s user 0.000s sys 0.010s
I20260806 03:43:35.549764 3334 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:35.071501 3710 consensus_meta.cc:287] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.193s user 0.009s sys 0.005s
I20260806 03:43:35.424312 3959 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:35.611686 659 raft_consensus.cc:2396] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 03:43:35.589001 3902 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.008s sys 0.016s
I20260806 03:43:35.508452 3862 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.600659 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:35.600832 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2238) took 1382 ms. Trace:
I20260806 03:43:35.616919 767 raft_consensus.cc:2396] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:35.594854 1658 rpcz_store.cc:276] 0806 03:43:34.207678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.207808 (+ 130us) service_pool.cc:227] Handling call
0806 03:43:34.208119 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:34.727872 (+519753us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:35.519642 (+791770us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54006fbb54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:35.519912 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":997377920}
I20260806 03:43:35.598451 3613 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:35.602736 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:35.603700 3647 consensus_meta.cc:287] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.829s user 0.010s sys 0.000s
I20260806 03:43:35.503572 3887 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.604777 3599 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:35.516343 3955 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:35.606179 1622 connection.cc:232] RPC call timeout handler was delayed by -1.1223706279415637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
I20260806 03:43:35.606905 1668 raft_consensus.cc:2362] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8. Re-sending same reply.
I20260806 03:43:35.595711 665 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:35.613044 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:35.615478 668 raft_consensus.cc:2396] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:35.587450 3906 consensus_queue.cc:1035] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260806 03:43:35.601194 3937 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:35.617180 663 rpcz_store.cc:276] 0806 03:43:34.218649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.218846 (+ 197us) service_pool.cc:227] Handling call
0806 03:43:34.219180 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:34.222490 (+ 3310us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:34.222518 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:34.228197 (+ 5679us) log.cc:842] Serialized 26 byte log entry
0806 03:43:34.454205 (+226008us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:34.454256 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:34.454352 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:35.600559 (+1146207us) raft_consensus.cc:1671] finished
0806 03:43:35.600585 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:35.600746 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2863,"raft.queue_time_us":237627,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":19317,"thread_start_us":225095,"threads_started":1,"wal-append.queue_time_us":311889}
I20260806 03:43:35.615401 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:35.554248 3876 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:35.587627 3948 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:35.590726 3518 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.475728 3960 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:35.614956 3902 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s user 0.008s sys 0.016s
I20260806 03:43:35.620446 3613 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:35.615980 3334 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.613996 3977 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.675597 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:35.624615 1622 connection.cc:232] RPC call timeout handler was delayed by -1.121470628073439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260806 03:43:35.674422 3814 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.614264 3961 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.613260 3922 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:35.787438 3862 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:35.613718 3975 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.772675 775 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:35.822882 3614 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:35.829481 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 291) took 1027 ms. Trace:
I20260806 03:43:35.822619 1622 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:35.823202 3814 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.788964 2911 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.735543 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:35.613648 3883 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:35.615607 3703 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.613857 3710 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.615831 3959 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:35.880872 1668 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:36.026912 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2139) took 1045 ms. Trace:
W20260806 03:43:36.044332 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 132125us
I20260806 03:43:36.020236 674 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:36.051461 665 rpcz_store.cc:276] 0806 03:43:34.801684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.801828 (+ 144us) service_pool.cc:227] Handling call
0806 03:43:35.829393 (+1027565us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:35.912915 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 752919us
I20260806 03:43:36.052443 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 290) took 1251 ms. Trace:
I20260806 03:43:36.033560 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.052947 674 rpcz_store.cc:276] 0806 03:43:34.801363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.801573 (+ 210us) service_pool.cc:227] Handling call
0806 03:43:36.019719 (+1218146us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bfb7c 00007f32133bf73b 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:36.019986 (+ 267us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:36.052194 (+ 32208us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.052390 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2461312}
I20260806 03:43:36.053969 659 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:35.913610 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.38512e+06us
I20260806 03:43:35.875099 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:35.756322 3978 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.057266 3978 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:36.057765 3901 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.995s user 0.000s sys 0.041s
W20260806 03:43:36.059192 3901 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.996s user 0.000s sys 0.042s
I20260806 03:43:36.062264 3978 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.078593 2911 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.081598 767 raft_consensus.cc:2379] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:35.933075 727 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:36.084271 3913 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.085368 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.085790 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.086516 3596 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.086716 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.087316 561 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:36.087042 3596 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:35.944878 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.051026 768 rpcz_store.cc:276] 0806 03:43:34.981994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.097332 (+115338us) service_pool.cc:227] Handling call
0806 03:43:35.179256 (+ 81924us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:35.586998 (+407742us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:35.587030 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:35.587084 (+ 54us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:35.587126 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.026627 (+439501us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54001031d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:36.026689 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:36.026835 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":937218304}
I20260806 03:43:36.120553 1668 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:36.123127 1670 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:36.119689 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.092733 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.091483 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 289) took 1303 ms. Trace:
I20260806 03:43:36.128397 665 raft_consensus.cc:2379] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:35.670116 3958 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.354s user 0.001s sys 0.000s
I20260806 03:43:36.128672 669 rpcz_store.cc:276] 0806 03:43:34.788356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.788522 (+ 166us) service_pool.cc:227] Handling call
0806 03:43:36.090841 (+1302319us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:36.091019 (+ 178us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:36.091194 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:36.091217 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:36.091251 (+ 34us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:36.091289 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.091338 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:36.091428 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":903693056}
I20260806 03:43:35.943872 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2136) took 1127 ms. Trace:
I20260806 03:43:36.090348 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.089962 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:35.676976 3986 log.cc:924] Time spent T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 03:43:36.130072 2911 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:35.789165 3940 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.016s user 0.011s sys 0.000s
I20260806 03:43:36.128604 771 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:36.191056 771 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:36.191193 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.136029 773 rpcz_store.cc:276] 0806 03:43:34.970280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.989287 (+ 19007us) service_pool.cc:227] Handling call
0806 03:43:35.019884 (+ 30597us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:35.064339 (+ 44455us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:35.064384 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:35.191674 (+127290us) log.cc:842] Serialized 26 byte log entry
0806 03:43:35.614288 (+422614us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:35.614316 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:35.614364 (+ 48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:35.943621 (+329257us) raft_consensus.cc:1671] finished
0806 03:43:35.943642 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:35.943798 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":68523,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":2053,"thread_start_us":283866,"threads_started":1,"wal-append.queue_time_us":621442}
W20260806 03:43:36.155921 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 222946us
I20260806 03:43:35.784940 3357 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:35.821141 3876 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.137501 563 raft_consensus.cc:2379] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:35.943892 3651 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.314s user 0.008s sys 0.012s
I20260806 03:43:36.139998 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.197702 772 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:36.197940 674 raft_consensus.cc:2396] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:36.186308 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.195050 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.198814 769 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:35.792737 3770 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:36.029630 3963 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.699s user 0.001s sys 0.001s
I20260806 03:43:35.852743 3922 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:36.010231 1658 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:36.199162 544 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:36.186398 776 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 03:43:36.200024 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 292) took 1371 ms. Trace:
W20260806 03:43:36.201160 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 65511.1us
I20260806 03:43:36.215067 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.217851 661 rpcz_store.cc:276] 0806 03:43:34.817433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.817585 (+ 152us) service_pool.cc:227] Handling call
0806 03:43:35.944689 (+1127104us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bfb7c 00007f32133bf73b 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:36.189203 (+244514us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1447989504}
W20260806 03:43:36.201344 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 380553us
I20260806 03:43:36.219712 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.220670 665 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:36.144497 546 raft_consensus.cc:2396] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:36.203197 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:36.203572 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:35.792968 3942 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:36.213092 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:35.983862 3604 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.359s user 0.016s sys 0.000s
I20260806 03:43:35.983973 3972 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:36.222764 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.089574 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.014289 3518 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.054999 3917 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:36.217196 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 95173.5us
I20260806 03:43:36.025941 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 350) took 2186 ms. Trace:
I20260806 03:43:36.221727 669 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:36.225589 1624 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:36.218974 771 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:36.089937 3874 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:36.226092 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.120096 3703 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.121304 3614 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.134521 3941 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.199s
I20260806 03:43:36.140220 3887 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:36.227731 659 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:35.623117 3647 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.220261 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:36.217722 776 pending_rounds.cc:76] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 5
I20260806 03:43:36.089037 3596 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.139191 3332 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260806 03:43:36.213975 3992 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:36.231465 3960 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.232777 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1265) took 1589 ms. Trace:
W20260806 03:43:36.233753 3994 log.cc:924] Time spent T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.415s user 0.001s sys 0.000s
I20260806 03:43:36.234819 3977 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.138355 3901 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:36.223451 2911 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:36.199007 3876 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:36.203686 3450 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:36.084894 3913 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.213287 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.120940 3975 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.213549 3651 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.191604 3940 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.479200 766 rpcz_store.cc:276] 0806 03:43:34.643409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.654433 (+ 11024us) service_pool.cc:227] Handling call
0806 03:43:34.739542 (+ 85109us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:34.816901 (+ 77359us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:34.827824 (+ 10923us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:34.980542 (+152718us) log.cc:842] Serialized 26 byte log entry
0806 03:43:35.319931 (+339389us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:35.324796 (+ 4865us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:35.386313 (+ 61517us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:36.232460 (+846147us) raft_consensus.cc:1671] finished
0806 03:43:36.232497 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:36.232684 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":171052,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":24020,"thread_start_us":241914,"threads_started":1,"wal-append.queue_time_us":487359}
I20260806 03:43:36.089516 3959 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:36.275753 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.237227 3955 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.473325 770 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:36.451808 542 raft_consensus.cc:2354] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:36.285048 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.476464 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:36.398468 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 48243.5us
I20260806 03:43:36.337152 3972 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:36.503887 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 278349us
I20260806 03:43:36.481469 1665 rpcz_store.cc:276] 0806 03:43:33.839910 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:33.847539 (+ 7629us) service_pool.cc:227] Handling call
0806 03:43:33.847946 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:34.695686 (+847740us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.025550 (+1329864us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400525654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:36.025822 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1668395776}
I20260806 03:43:36.473444 776 pending_rounds.cc:97] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 03:43:36.482591 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:36.483340 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 141322us
I20260806 03:43:36.491074 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1694) took 1004 ms. Trace:
I20260806 03:43:36.491271 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.214227 3357 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.275776 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.475684 3948 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.475051 661 raft_consensus.cc:2354] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 5.
W20260806 03:43:36.502442 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 89396.8us
I20260806 03:43:36.508708 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:36.510916 669 rpcz_store.cc:276] 0806 03:43:35.486469 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.136061 (+649592us) service_pool.cc:227] Handling call
0806 03:43:36.490997 (+354936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.372819 3961 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.276844 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.273517 3604 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.497265 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.487308 3332 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:36.403636 3980 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:36.438642 3937 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.148s
I20260806 03:43:36.273202 3942 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.353349 3902 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.255s
I20260806 03:43:36.270828 3770 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.511685 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.737018 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.737584 1670 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:36.737867 771 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:36.738868 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.739045 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:36.518910 3862 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:36.482769 3450 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:36.740453 769 raft_consensus.cc:2396] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:36.739840 771 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:43:36.510591 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.529136 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.813730 561 raft_consensus.cc:2379] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:36.813979 1668 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:36.615048 667 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:36.510128 668 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:36.516723 3975 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.820006 3703 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.524453 671 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:36.812378 3450 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:36.815375 3960 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:36.821484 3960 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.821874 3960 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:36.821923 671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1683) took 1339 ms (client timeout 1527 ms). Trace:
W20260806 03:43:36.822666 671 rpcz_store.cc:269] 0806 03:43:35.482650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.564663 (+ 82013us) service_pool.cc:227] Handling call
0806 03:43:36.821734 (+1257071us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54002cd654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:36.821865 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2204377344}
I20260806 03:43:36.823822 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.824542 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2251) took 1277 ms. Trace:
I20260806 03:43:36.824843 671 rpcz_store.cc:276] 0806 03:43:35.546786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.823512 (+1276726us) service_pool.cc:227] Handling call
0806 03:43:36.824493 (+ 981us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.823503 3450 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:36.588066 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 54516.2us
I20260806 03:43:36.644876 3941 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:36.795400 3647 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.830690 3941 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.831331 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.832665 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.833165 3941 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:36.815925 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.833967 548 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:36.515317 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.773765 3940 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.475909 3874 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.384181 3883 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.835878 614 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:36.836126 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:36.836905 766 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:36.837069 614 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:36.516871 4007 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.588124 3913 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.518666 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.771539 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.838521 3913 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.839138 3913 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.840824 1658 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:36.841231 3913 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.818790 3334 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.813689 776 raft_consensus.cc:2866] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:36.818150 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.844027 669 raft_consensus.cc:2396] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:43:36.844106 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.500701 3901 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:36.811707 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 303113us
I20260806 03:43:36.817915 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1693) took 1331 ms. Trace:
I20260806 03:43:36.854759 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:36.855093 661 rpcz_store.cc:276] 0806 03:43:35.486052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.225752 (+739700us) service_pool.cc:227] Handling call
0806 03:43:36.817851 (+592099us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:36.856247 4003 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260806 03:43:36.825874 3450 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:36.834551 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:36.833223 775 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:36.839417 1657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:55938
I20260806 03:43:36.857425 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 715) took 1395 ms. Trace:
I20260806 03:43:36.857975 1657 raft_consensus.cc:595] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:36.858517 1657 raft_consensus.cc:603] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:36.857942 1662 rpcz_store.cc:276] 0806 03:43:35.461813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.473091 (+ 11278us) service_pool.cc:227] Handling call
0806 03:43:35.481525 (+ 8434us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:35.542506 (+ 60981us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:35.588604 (+ 46098us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:35.675016 (+ 86412us) log.cc:842] Serialized 26 byte log entry
0806 03:43:36.340145 (+665129us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:36.343528 (+ 3383us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.352599 (+ 9071us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:36.857187 (+504588us) raft_consensus.cc:1671] finished
0806 03:43:36.857203 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:36.857357 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38138,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":72,"thread_start_us":426020,"threads_started":2,"wal-append.queue_time_us":771886}
I20260806 03:43:36.586969 3959 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:36.772732 3651 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.518137 4000 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.347s
I20260806 03:43:36.860460 3959 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.860898 3959 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:36.818629 3980 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.862246 769 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:36.710948 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1268) took 2005 ms. Trace:
I20260806 03:43:36.842272 4007 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.863312 770 rpcz_store.cc:276] 0806 03:43:34.705491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.705752 (+ 261us) service_pool.cc:227] Handling call
0806 03:43:35.779313 (+1073561us) spinlock_profiling.cc:243] Waited 9.14 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:36.473117 (+693804us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:36.496550 (+ 23433us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.710615 (+214065us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002bd9d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:36.710846 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616105216}
I20260806 03:43:36.864935 3651 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:36.842787 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2245) took 1363 ms. Trace:
I20260806 03:43:36.843415 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.866137 667 rpcz_store.cc:276] 0806 03:43:35.479569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.479698 (+ 129us) service_pool.cc:227] Handling call
0806 03:43:36.838313 (+1358615us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540019bed4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:36.842723 (+ 4410us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2339842816}
W20260806 03:43:36.804849 664 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.984s user 0.012s sys 0.000s
I20260806 03:43:36.867913 664 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:36.829744 3960 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.515317 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.868645 1668 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:36.820369 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1696) took 1333 ms. Trace:
I20260806 03:43:36.844645 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1685) took 1361 ms. Trace:
I20260806 03:43:36.596027 3955 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:36.846073 4022 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:36.846611 3940 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:36.844960 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 714) took 1401 ms. Trace:
I20260806 03:43:36.849072 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:36.849838 3874 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.850476 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.829746 501 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.850838 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.851384 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:36.851635 3961 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.852111 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.852178 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.894163 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.852417 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:36.894778 563 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:36.894948 562 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:36.852893 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:36.896097 663 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:36.853058 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:36.853176 771 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:36.889323 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:43:36.897619 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.856103 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.899325 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1706) took 1075 ms. Trace:
I20260806 03:43:36.899742 661 rpcz_store.cc:276] 0806 03:43:35.823817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.855863 (+1032046us) service_pool.cc:227] Handling call
0806 03:43:36.899249 (+ 43386us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:36.901470 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 71807.5us
W20260806 03:43:36.806468 4012 log.cc:924] Time spent T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260806 03:43:36.877307 3933 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.877003 3980 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.817898 3942 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.869594 669 rpcz_store.cc:276] 0806 03:43:35.483347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.817827 (+1334480us) service_pool.cc:227] Handling call
0806 03:43:36.844585 (+ 26758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.518349 3972 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:36.932720 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 83774.2us
I20260806 03:43:36.869307 668 rpcz_store.cc:276] 0806 03:43:35.487263 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.482178 (+994915us) service_pool.cc:227] Handling call
0806 03:43:36.820308 (+338130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.749863 499 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.666849 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.842294 1658 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:43:36.589921 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.591311 1664 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:36.515810 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1697) took 1028 ms. Trace:
I20260806 03:43:36.942652 674 rpcz_store.cc:276] 0806 03:43:35.487596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.225120 (+737524us) service_pool.cc:227] Handling call
0806 03:43:36.515745 (+290625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.878934 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:36.883497 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 663273us
I20260806 03:43:36.882097 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1687) took 1398 ms. Trace:
I20260806 03:43:36.945093 659 rpcz_store.cc:276] 0806 03:43:35.484003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.491918 (+1007915us) service_pool.cc:227] Handling call
0806 03:43:36.882035 (+390117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.888051 544 raft_consensus.cc:2354] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 4.
I20260806 03:43:36.887797 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:36.888505 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.888724 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.946291 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.947707 667 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:36.882803 1671 rpcz_store.cc:276] 0806 03:43:35.443203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.443306 (+ 103us) service_pool.cc:227] Handling call
0806 03:43:35.472482 (+ 29176us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:36.033786 (+561304us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:36.033817 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:36.120091 (+ 86274us) log.cc:842] Serialized 26 byte log entry
0806 03:43:36.407892 (+287801us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:36.407944 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.408041 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:36.844771 (+436730us) raft_consensus.cc:1671] finished
0806 03:43:36.844790 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:36.844902 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46912,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":608722,"threads_started":2,"wal-append.queue_time_us":609750}
I20260806 03:43:36.510064 3710 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.669831 1660 raft_consensus.cc:1226] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:36.894256 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1704) took 1071 ms. Trace:
I20260806 03:43:36.950922 671 rpcz_store.cc:276] 0806 03:43:35.823225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.850033 (+1026808us) service_pool.cc:227] Handling call
0806 03:43:36.894181 (+ 44148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.900897 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.905810 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.632750 3887 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:36.898471 771 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:36.857461 3901 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.671340 3814 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.927480 3647 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:36.883196 3955 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.769929 3357 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.902144 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:36.758924 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:36.868774 664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1701) took 1260 ms (client timeout 1570 ms). Trace:
W20260806 03:43:36.978605 664 rpcz_store.cc:269] 0806 03:43:35.608251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.619370 (+ 11119us) service_pool.cc:227] Handling call
0806 03:43:36.868714 (+1249344us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.979578 544 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:36.980208 664 raft_consensus.cc:1226] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:43:36.927712 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.720899 3518 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.199s
I20260806 03:43:36.935588 665 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:36.981657 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1707) took 1157 ms. Trace:
I20260806 03:43:36.982053 665 rpcz_store.cc:276] 0806 03:43:35.824092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.560680 (+736588us) service_pool.cc:227] Handling call
0806 03:43:36.981582 (+420902us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.694952 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.890620 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:36.913309 3874 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:36.981009 558 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:36.986891 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:36.942118 3940 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:43:36.936340 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 186598us
I20260806 03:43:37.008985 499 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.957168 661 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:37.009793 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1682) took 1527 ms. Trace:
I20260806 03:43:37.010114 661 rpcz_store.cc:276] 0806 03:43:35.482249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.900573 (+1418324us) service_pool.cc:227] Handling call
0806 03:43:37.009737 (+109164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.959437 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:37.013077 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53773.8us
I20260806 03:43:36.960973 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:36.518322 3992 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:36.938107 548 raft_consensus.cc:2354] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4.
I20260806 03:43:36.884564 3948 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.962527 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2247) took 1453 ms. Trace:
I20260806 03:43:37.027443 659 rpcz_store.cc:276] 0806 03:43:35.509346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.945962 (+1436616us) service_pool.cc:227] Handling call
0806 03:43:36.962431 (+ 16469us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.968384 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:36.968883 775 raft_consensus.cc:1226] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:36.970587 663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1695) took 1483 ms (client timeout 1799 ms). Trace:
I20260806 03:43:36.973325 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:37.029711 663 rpcz_store.cc:269] 0806 03:43:35.486872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.868524 (+381652us) service_pool.cc:227] Handling call
0806 03:43:36.970390 (+1101866us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b540036ddd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:36.970523 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1929434368}
I20260806 03:43:36.973964 776 raft_consensus.cc:1261] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:36.975286 767 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:36.976864 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:37.032328 668 raft_consensus.cc:2379] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:36.891124 3332 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.976722 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.987471 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1692) took 1501 ms. Trace:
W20260806 03:43:36.954095 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48361.4us
I20260806 03:43:36.986313 671 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:37.040392 664 rpcz_store.cc:276] 0806 03:43:35.485732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.979493 (+1493761us) service_pool.cc:227] Handling call
0806 03:43:36.979919 (+ 426us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:36.987143 (+ 7224us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:36.987399 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:36.985690 3955 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:36.990908 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:36.991832 542 raft_consensus.cc:2354] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:36.992050 3874 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.966214 4024 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:43:36.966670 3905 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.039s sys 0.077s
I20260806 03:43:36.994711 3917 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 03:43:36.995731 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 61085.1us
I20260806 03:43:37.077495 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.078217 558 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:37.078359 1670 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:37.002619 3604 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.933312 3933 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:37.085321 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:37.087188 3332 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:37.087491 3933 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:37.087821 3332 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:37.037339 3992 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:36.963325 3902 consensus_queue.cc:1046] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260806 03:43:37.090019 3948 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:36.950613 3878 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:37.092036 1668 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:37.004591 4007 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:36.906630 3961 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:37.097693 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:36.936720 4010 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
W20260806 03:43:37.070845 3613 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.218s user 0.011s sys 0.003s
I20260806 03:43:37.011705 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2249) took 1482 ms. Trace:
I20260806 03:43:37.104398 661 rpcz_store.cc:276] 0806 03:43:35.529219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.010728 (+1481509us) service_pool.cc:227] Handling call
0806 03:43:37.011051 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.011285 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:37.011316 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:37.011371 (+ 55us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:37.011434 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:37.011501 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:37.011636 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:37.034065 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 60858.8us
I20260806 03:43:37.106782 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:37.036580 3975 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:37.037317 499 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:37.040897 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1689) took 1556 ms. Trace:
I20260806 03:43:36.969189 3887 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:37.097380 3878 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:37.052188 501 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:36.993592 665 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:36.978049 3614 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:37.065565 562 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:37.071301 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:37.023957 3940 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:37.073455 3905 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s user 0.039s sys 0.078s
I20260806 03:43:37.005908 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.080946 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:37.028985 3913 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:37.030339 3942 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:37.066774 3874 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:37.085248 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:37.085395 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:37.049981 3944 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:37.005681 3960 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:37.091706 3922 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.856s user 0.005s sys 0.005s
I20260806 03:43:37.069676 4024 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:36.989074 3647 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:36.934427 4022 consensus_queue.cc:1046] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.132s
I20260806 03:43:37.051205 669 raft_consensus.cc:2396] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:37.118556 671 rpcz_store.cc:276] 0806 03:43:35.484682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.951827 (+1467145us) service_pool.cc:227] Handling call
0806 03:43:36.986156 (+ 34329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.040655 (+ 54499us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:37.040835 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:37.267171 667 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:37.277262 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.279639 663 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:37.280458 558 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:37.280411 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2246) took 1779 ms. Trace:
W20260806 03:43:37.280675 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 195371us
I20260806 03:43:37.248520 544 raft_consensus.cc:2354] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:37.281212 667 rpcz_store.cc:276] 0806 03:43:35.501003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.867330 (+1366327us) service_pool.cc:227] Handling call
0806 03:43:37.280338 (+413008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:37.282745 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:37.122797 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 85567us
I20260806 03:43:37.285143 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.282717 776 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
W20260806 03:43:37.285579 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 233515us
I20260806 03:43:37.286552 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1686) took 1802 ms. Trace:
I20260806 03:43:37.285902 546 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:37.287012 669 rpcz_store.cc:276] 0806 03:43:35.483675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.936376 (+1452701us) service_pool.cc:227] Handling call
0806 03:43:37.286437 (+350061us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:37.278244 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 171530us
I20260806 03:43:37.288583 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:37.279898 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2248) took 1764 ms. Trace:
I20260806 03:43:37.297652 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.298604 1664 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:37.300149 615 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:37.289263 665 rpcz_store.cc:276] 0806 03:43:35.515883 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.982875 (+1466992us) service_pool.cc:227] Handling call
0806 03:43:36.993227 (+ 10352us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.279668 (+286441us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:37.279841 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:37.308439 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:37.093214 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 579) took 1650 ms. Trace:
I20260806 03:43:37.314550 563 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:37.309283 775 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 1.
I20260806 03:43:37.118176 3975 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:37.137677 4007 raft_consensus.cc:3032] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:37.332044 1624 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:37.317783 1668 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:37.139436 3862 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:37.335911 727 leader_election.cc:398] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 5.
I20260806 03:43:37.185068 3703 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:37.290689 4024 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:37.288434 3770 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:37.289342 3940 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:37.328014 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 288727us
I20260806 03:43:37.246804 3604 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:37.282833 3942 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:37.285430 3878 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:37.335429 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.301322 3906 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:37.338505 771 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:37.296160 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.341208 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:37.341526 563 raft_consensus.cc:1226] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20260806 03:43:37.301064 4015 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:37.295733 4029 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:37.349275 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:37.137065 3902 consensus_queue.cc:1046] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.195s
I20260806 03:43:37.331239 3944 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:37.316922 4022 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:37.329417 3972 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:37.331775 3975 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:37.366081 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:37.379210 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.379561 3906 raft_consensus.cc:686] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:37.380157 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:37.381234 4027 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20260806 03:43:37.332044 4007 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:37.319927 1660 rpcz_store.cc:276] 0806 03:43:35.442588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:35.460908 (+ 18320us) service_pool.cc:227] Handling call
0806 03:43:35.483028 (+ 22120us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:36.969081 (+1486053us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:37.059536 (+ 90455us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54004f0454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:37.082661 (+ 23125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2387874176}
I20260806 03:43:37.351902 3942 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:37.346169 542 raft_consensus.cc:1226] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
W20260806 03:43:37.385944 769 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.523s user 0.007s sys 0.000s
W20260806 03:43:37.388558 4025 log.cc:924] Time spent T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260806 03:43:37.394532 548 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:37.396680 3874 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:37.403591 3887 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:37.404745 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 93484.5us
I20260806 03:43:37.496920 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:37.121328 3613 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:37.417071 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 67917.4us
I20260806 03:43:37.360832 1665 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:37.353621 614 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:37.297283 3913 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:37.296972 3922 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:37.366323 3604 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:37.342790 3960 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:37.448493 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.452724 546 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:37.352625 3450 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:37.456650 563 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 03:43:37.419047 3980 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:37.421922 3614 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:37.483398 769 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:37.446108 4022 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:37.454489 766 consensus_meta.cc:287] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.617s user 0.010s sys 0.000s
I20260806 03:43:37.446478 3972 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:37.495442 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.456863 3941 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:37.424643 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:37.510794 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:37.511510 4049 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260806 03:43:37.513475 3906 consensus_queue.cc:237] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:37.516010 3961 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:37.412691 4015 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:37.522912 727 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:37.548295 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:38.246898 563 pending_rounds.cc:76] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 5
W20260806 03:43:37.948096 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.049s sys 0.070s
W20260806 03:43:38.033349 772 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.045s sys 0.135s
I20260806 03:43:38.264740 563 pending_rounds.cc:97] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 03:43:38.265189 563 raft_consensus.cc:2866] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:43:38.045141 1662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.020s sys 0.012s
W20260806 03:43:38.265985 1662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s user 0.026s sys 0.042s
W20260806 03:43:38.025213 768 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.031s user 0.049s sys 0.093s
I20260806 03:43:38.266675 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:38.267308 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1800) took 1220 ms. Trace:
W20260806 03:43:38.267366 768 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.273s user 0.070s sys 0.113s
I20260806 03:43:38.267691 544 rpcz_store.cc:276] 0806 03:43:37.046885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.068107 (+ 21222us) service_pool.cc:227] Handling call
0806 03:43:38.267138 (+1199031us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54001c84d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:38.267256 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":372290944}
I20260806 03:43:38.268148 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:37.694684 3609 raft_consensus.cc:3032] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:37.941395 767 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.025s sys 0.065s
W20260806 03:43:38.264808 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.974s user 0.084s sys 0.135s
W20260806 03:43:38.243530 3357 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.008s sys 0.036s
W20260806 03:43:38.270061 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 985444us
W20260806 03:43:38.270551 3357 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.008s sys 0.039s
W20260806 03:43:37.855089 3905 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.075s sys 0.165s
W20260806 03:43:38.271515 3905 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.942s user 0.165s sys 0.257s
I20260806 03:43:38.271330 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:37.628582 3922 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:38.265057 772 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.924s user 0.059s sys 0.160s
W20260806 03:43:38.118827 3814 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s user 0.000s sys 0.011s
I20260806 03:43:38.273010 499 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:37.631423 3874 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.266641 3980 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.274505 3972 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:38.274538 499 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:38.274806 562 raft_consensus.cc:1226] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260806 03:43:38.119490 3992 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.028s user 0.041s sys 0.071s
W20260806 03:43:38.276330 3814 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s user 0.000s sys 0.012s
W20260806 03:43:38.276669 3992 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.186s user 0.041s sys 0.072s
I20260806 03:43:37.696192 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:38.278389 3814 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:37.594132 558 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:38.278537 3941 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.279606 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1802) took 1195 ms. Trace:
I20260806 03:43:38.279501 3814 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:38.280148 558 rpcz_store.cc:276] 0806 03:43:37.084163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.084397 (+ 234us) service_pool.cc:227] Handling call
0806 03:43:38.279545 (+1195148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.280967 3941 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:38.269973 767 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s user 0.046s sys 0.100s
I20260806 03:43:37.595516 4007 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.272922 3922 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:38.243857 773 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.002s sys 0.010s
W20260806 03:43:38.290851 773 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.003s sys 0.011s
I20260806 03:43:38.275398 3980 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.282204 1664 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:38.291718 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1286) took 1409 ms. Trace:
I20260806 03:43:37.579361 1657 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260806 03:43:38.292908 4054 log.cc:924] Time spent T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.721s user 0.000s sys 0.013s
I20260806 03:43:38.283102 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1283) took 1462 ms. Trace:
I20260806 03:43:38.294100 769 rpcz_store.cc:276] 0806 03:43:36.821016 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.834280 (+ 13264us) service_pool.cc:227] Handling call
0806 03:43:38.283017 (+1448737us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.295476 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:38.297492 769 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:43:38.288560 3901 raft_consensus.cc:3032] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:37.629956 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:38.079466 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.006s user 0.003s sys 0.008s
W20260806 03:43:38.305058 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.232s user 0.004s sys 0.008s
W20260806 03:43:38.305411 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.24378e+06us
I20260806 03:43:38.282802 3814 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:38.150215 3883 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.003s sys 0.008s
W20260806 03:43:38.093375 728 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.005s sys 0.003s
W20260806 03:43:38.183974 3878 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.823s user 0.011s sys 0.000s
I20260806 03:43:37.736702 3647 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:37.889957 3604 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:38.103607 3613 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:38.157972 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.004s sys 0.004s
W20260806 03:43:38.324653 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.004s sys 0.005s
W20260806 03:43:37.578924 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 233338us
I20260806 03:43:38.285688 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 467) took 1194 ms. Trace:
I20260806 03:43:38.325266 769 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:43:38.079041 1668 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.005s sys 0.004s
I20260806 03:43:38.274290 3874 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260806 03:43:38.325464 771 rpcz_store.cc:276] 0806 03:43:37.091724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.097373 (+ 5649us) service_pool.cc:227] Handling call
0806 03:43:38.285353 (+1187980us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400186354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:38.285626 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":552105216}
I20260806 03:43:37.524195 3975 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.283162 766 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:38.013495 3876 consensus_meta.cc:287] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.537s user 0.009s sys 0.004s
I20260806 03:43:38.283908 4022 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:38.239488 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 715504us
I20260806 03:43:38.334968 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:37.524199 3887 consensus_queue.cc:1046] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.759s
I20260806 03:43:38.287137 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2250) took 2747 ms. Trace:
I20260806 03:43:38.292475 773 rpcz_store.cc:276] 0806 03:43:36.881973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.005147 (+123174us) service_pool.cc:227] Handling call
0806 03:43:37.005422 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.005634 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:43:37.291048 (+285414us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.291317 (+1000269us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.291388 (+ 71us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:38.291447 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.291524 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.291656 (+ 132us) inbound_call.cc:173] Queueing success response
0806 03:43:38.292289 (+ 633us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
Metrics: {"prepare.queue_time_us":505165,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":597237,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":53174016,"thread_start_us":1284757,"threads_started":3,"wal-append.queue_time_us":232387}
I20260806 03:43:38.209828 4050 raft_consensus.cc:3032] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:38.268883 563 raft_consensus.cc:2379] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:37.549697 3960 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:38.289417 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1708) took 1545 ms. Trace:
I20260806 03:43:38.346890 663 rpcz_store.cc:276] 0806 03:43:36.743519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.042922 (+299403us) service_pool.cc:227] Handling call
0806 03:43:37.279450 (+236528us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.280098 (+ 648us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.289198 (+1009100us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400387050. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:38.289355 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":543190272}
W20260806 03:43:38.252812 1670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 726) took 1561 ms (client timeout 1641 ms). Trace:
W20260806 03:43:38.348165 1670 rpcz_store.cc:269] 0806 03:43:36.691146 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.738602 (+ 47456us) service_pool.cc:227] Handling call
0806 03:43:38.252553 (+1513951us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400529754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:38.252731 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":779232128}
W20260806 03:43:37.880959 3959 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.020s user 0.019s sys 0.001s
W20260806 03:43:38.312403 728 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.006s sys 0.003s
I20260806 03:43:38.262501 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:38.284225 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 761348us
I20260806 03:43:38.269335 3609 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:38.298161 3980 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:38.282342 3450 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:38.116724 659 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.062s user 0.001s sys 0.015s
W20260806 03:43:38.326195 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 819398us
W20260806 03:43:38.326362 1668 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s user 0.007s sys 0.004s
I20260806 03:43:38.361915 548 raft_consensus.cc:2354] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4.
W20260806 03:43:38.387265 659 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.333s user 0.001s sys 0.023s
I20260806 03:43:38.268864 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:38.303906 3901 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:38.310899 3883 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s user 0.003s sys 0.008s
I20260806 03:43:38.329751 1624 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:38.285962 3941 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:38.309255 542 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:38.317646 3332 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:38.337462 674 rpcz_store.cc:276] 0806 03:43:35.540168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.023656 (+1483488us) service_pool.cc:227] Handling call
0806 03:43:37.281251 (+257595us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.281421 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:37.281443 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:37.281482 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:37.281548 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:37.281609 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.286929 (+1005320us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540036ced0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:38.287052 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":523139456}
I20260806 03:43:38.342659 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:38.492058 771 raft_consensus.cc:2396] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:38.256722 3710 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.087s
I20260806 03:43:38.369623 614 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:38.364068 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1278) took 1845 ms. Trace:
I20260806 03:43:38.387959 544 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:38.361415 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 868224us
I20260806 03:43:38.361934 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:38.339666 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1284) took 1515 ms. Trace:
I20260806 03:43:38.443320 4048 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:38.327627 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:38.505712 769 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:38.366473 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2170) took 1056 ms. Trace:
W20260806 03:43:38.506316 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 144484us
I20260806 03:43:38.506578 773 rpcz_store.cc:276] 0806 03:43:37.310184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.345478 (+1035294us) service_pool.cc:227] Handling call
0806 03:43:38.365795 (+ 20317us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.366044 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:38.366082 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.366135 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:38.366184 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.366249 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.366400 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.340437 4050 raft_consensus.cc:729] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:38.434079 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2152) took 2231 ms. Trace:
W20260806 03:43:38.427307 546 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.844s user 0.009s sys 0.014s
I20260806 03:43:38.337148 3876 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:38.360140 3940 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.944s user 0.009s sys 0.002s
I20260806 03:43:38.515187 768 rpcz_store.cc:276] 0806 03:43:36.195481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.215894 (+ 20413us) service_pool.cc:227] Handling call
0806 03:43:36.216215 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:36.235083 (+ 18868us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:36.235120 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:36.470141 (+235021us) log.cc:842] Serialized 26 byte log entry
0806 03:43:36.966302 (+496161us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:38.268074 (+1301772us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.426342 (+158268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.426449 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:38.426496 (+ 47us) raft_consensus.cc:1671] finished
0806 03:43:38.426506 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.426653 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3913,"prepare.queue_time_us":544183,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":32142,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":1926347,"threads_started":3,"wal-append.queue_time_us":786010}
I20260806 03:43:38.350479 3933 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:38.349730 3960 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:38.368256 3980 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:38.450598 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 172969us
I20260806 03:43:38.524926 769 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
W20260806 03:43:38.477337 3942 consensus_meta.cc:287] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.972s user 0.006s sys 0.006s
I20260806 03:43:38.479003 3901 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:38.360666 3907 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:38.368451 3992 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:38.480365 4059 log.cc:924] Time spent T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260806 03:43:38.428683 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:38.363427 4028 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 1.045s user 0.008s sys 0.009s
W20260806 03:43:38.483120 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 153456us
I20260806 03:43:38.536023 3933 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:38.366856 3959 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.496723 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:38.359745 4060 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
I20260806 03:43:38.493899 3614 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:38.501813 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2171) took 1175 ms. Trace:
I20260806 03:43:38.493773 770 rpcz_store.cc:276] 0806 03:43:36.518667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.899837 (+381170us) service_pool.cc:227] Handling call
0806 03:43:36.928907 (+ 29070us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:36.929184 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:43:37.418236 (+489052us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.363531 (+945295us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.363583 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:38.363781 (+ 198us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.363890 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.364012 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":557166,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":391286,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":1411168,"threads_started":3}
I20260806 03:43:38.324555 3878 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:38.513604 1622 connection.cc:232] RPC call timeout handler was delayed by -1.9425814559217542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17880000000000001s for a response
I20260806 03:43:38.519019 727 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:38.516481 546 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:38.614086 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1810) took 1272 ms. Trace:
W20260806 03:43:38.522800 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 215760us
W20260806 03:43:38.502810 4024 consensus_meta.cc:287] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.142s user 0.010s sys 0.003s
I20260806 03:43:38.527444 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2172) took 1140 ms. Trace:
I20260806 03:43:38.615869 769 rpcz_store.cc:276] 0806 03:43:37.387108 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.526178 (+1139070us) service_pool.cc:227] Handling call
0806 03:43:38.526792 (+ 614us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.527029 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:38.527066 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.527123 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:38.527176 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.527244 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.527372 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.528064 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:38.532572 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 469) took 1164 ms. Trace:
W20260806 03:43:38.534929 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 165410us
I20260806 03:43:38.529304 3710 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:38.530467 3940 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:38.498049 3609 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:38.624547 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 96584.1us
I20260806 03:43:38.625625 499 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:38.537639 3450 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:38.539014 544 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:38.630687 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 315) took 1138 ms. Trace:
I20260806 03:43:38.631168 544 rpcz_store.cc:276] 0806 03:43:37.492526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.268610 (+776084us) service_pool.cc:227] Handling call
0806 03:43:38.630602 (+361992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.543238 542 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:38.553194 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 470) took 1170 ms. Trace:
I20260806 03:43:38.556838 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:38.635300 768 rpcz_store.cc:276] 0806 03:43:37.383268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.522159 (+1138891us) service_pool.cc:227] Handling call
0806 03:43:38.552627 (+ 30468us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.552836 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:38.552860 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.552899 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:38.552936 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.552989 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.553128 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:38.563990 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 135397us
I20260806 03:43:38.566466 561 raft_consensus.cc:2354] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 4.
I20260806 03:43:38.566540 3901 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:38.527204 3941 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:38.443490 3913 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.648373 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:38.649072 769 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:38.603397 1664 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:38.571483 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:38.529570 3980 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:38.594714 1670 raft_consensus.cc:2396] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:38.604187 3959 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:38.604403 3933 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.602578 4057 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:38.603276 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:38.605832 3876 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:38.597203 3955 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.524s user 0.006s sys 0.006s
I20260806 03:43:38.498196 3332 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.551323 3942 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.592965 4050 consensus_queue.cc:260] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:38.517634 4048 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.612246 771 rpcz_store.cc:276] 0806 03:43:37.326052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.500891 (+1174839us) service_pool.cc:227] Handling call
0806 03:43:38.501285 (+ 394us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.501485 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:38.501512 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.501556 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:38.501597 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.501650 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.501752 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:38.613641 1622 connection.cc:232] RPC call timeout handler was delayed by -1.9029814559035003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
I20260806 03:43:38.498448 766 rpcz_store.cc:276] 0806 03:43:36.822012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.835843 (+ 13831us) service_pool.cc:227] Handling call
0806 03:43:38.337341 (+1501498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.614441 546 rpcz_store.cc:276] 0806 03:43:37.342153 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.342456 (+ 303us) service_pool.cc:227] Handling call
0806 03:43:38.614024 (+1271568us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.617792 773 rpcz_store.cc:276] 0806 03:43:37.368342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.508138 (+1139796us) service_pool.cc:227] Handling call
0806 03:43:38.531903 (+ 23765us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.532202 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:38.532230 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.532284 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:38.532327 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.532384 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.532505 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:38.570675 3975 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.661307 3609 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:38.630362 3710 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.538687 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:38.638098 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:38.640192 614 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:38.641569 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 134496us
I20260806 03:43:38.613416 3614 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:38.557603 3647 consensus_queue.cc:1159] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ada5ef6656cb4420baf77a4b4342909e" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:43:38.572932 3992 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:38.634362 4000 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:38.630277 3906 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:38.660753 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:38.623485 4024 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:38.662766 4090 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260806 03:43:38.642050 4079 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:38.650347 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 311) took 1188 ms. Trace:
W20260806 03:43:38.664593 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61406.1us
W20260806 03:43:38.786353 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 148380us
W20260806 03:43:38.705587 1622 connection.cc:232] RPC call timeout handler was delayed by -1.8930814559571445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
I20260806 03:43:38.707765 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 465) took 1703 ms. Trace:
W20260806 03:43:38.787164 1622 connection.cc:232] RPC call timeout handler was delayed by -1.7049814560450613s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260806 03:43:38.787564 1622 connection.cc:232] RPC call timeout handler was delayed by -1.6059814558830112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260806 03:43:38.787979 1622 connection.cc:232] RPC call timeout handler was delayed by -1.5411814560648054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260806 03:43:38.787240 767 rpcz_store.cc:276] 0806 03:43:37.004733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.052800 (+ 48067us) service_pool.cc:227] Handling call
0806 03:43:37.079585 (+ 26785us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:37.080032 (+ 447us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:37.080105 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:37.093294 (+ 13189us) log.cc:842] Serialized 26 byte log entry
0806 03:43:38.282005 (+1188711us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:38.282050 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.282126 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:38.707552 (+425426us) raft_consensus.cc:1671] finished
0806 03:43:38.707579 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.707707 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":682,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":2841,"thread_start_us":1004210,"threads_started":1,"wal-append.queue_time_us":1492229}
W20260806 03:43:38.713807 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 73746.9us
I20260806 03:43:38.714660 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:38.749593 501 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:38.750392 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2174) took 1155 ms. Trace:
W20260806 03:43:38.659408 4082 log.cc:924] Time spent T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
W20260806 03:43:38.698911 4065 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260806 03:43:38.782447 1670 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:38.789860 770 rpcz_store.cc:276] 0806 03:43:37.594698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.612616 (+1017918us) service_pool.cc:227] Handling call
0806 03:43:38.629131 (+ 16515us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.629409 (+ 278us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:38.651074 (+ 21665us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.749760 (+ 98686us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.750004 (+ 244us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.750056 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:38.750100 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.750168 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.750295 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":27505,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":238,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":38646,"thread_start_us":93103,"threads_started":1}
W20260806 03:43:38.754359 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 139139us
I20260806 03:43:38.792313 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:38.746222 3614 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:38.792853 1622 leader_election.cc:334] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.788s (SENT)
I20260806 03:43:38.711561 3959 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:38.700979 3942 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.670855 4061 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:38.793395 1622 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:38.705289 3940 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:38.699594 3332 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.700744 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:38.803046 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:38.703706 4048 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:38.806970 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 57476.2us
I20260806 03:43:38.848678 773 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:38.807114 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 92555.6us
I20260806 03:43:38.850070 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:38.752979 3883 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:38.753583 3905 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:38.782426 4000 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:38.801128 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:38.808609 564 raft_consensus.cc:2354] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:38.800613 3876 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:38.839540 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:38.868894 769 raft_consensus.cc:2396] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:38.839764 3940 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:43:38.860069 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 67243.4us
I20260806 03:43:38.839162 3874 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:43:38.882265 1622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.832s (SENT) [suppressed 17 similar messages]
I20260806 03:43:38.806802 3613 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:38.785966 542 rpcz_store.cc:276] 0806 03:43:37.456338 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.629541 (+173203us) service_pool.cc:227] Handling call
0806 03:43:38.644216 (+1014675us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:38.862216 3972 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.587s user 0.006s sys 0.012s
I20260806 03:43:38.710522 3955 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:38.922865 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 64169.2us
I20260806 03:43:38.859722 4000 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:38.943687 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:38.867822 4085 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:38.892357 3940 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:38.855001 3332 consensus_queue.cc:1046] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
W20260806 03:43:38.947125 4063 log.cc:924] Time spent T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
W20260806 03:43:38.947038 4093 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
I20260806 03:43:38.926294 3878 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:38.926538 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 356) took 2057 ms. Trace:
I20260806 03:43:38.996706 547 raft_consensus.cc:2354] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:38.966730 3940 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:39.067158 4024 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:38.860805 3883 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:39.073406 727 connection.cc:232] RPC call timeout handler was delayed by -2.1315624248236418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
W20260806 03:43:39.073945 727 connection.cc:232] RPC call timeout handler was delayed by -2.1135624249000102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260806 03:43:39.074774 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 362) took 1714 ms. Trace:
I20260806 03:43:38.859681 3905 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:39.074290 727 connection.cc:232] RPC call timeout handler was delayed by -2.0739624248817563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
W20260806 03:43:39.082934 727 connection.cc:232] RPC call timeout handler was delayed by -2.0325624248944223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260806 03:43:39.083321 727 connection.cc:232] RPC call timeout handler was delayed by -1.9983624247834086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
I20260806 03:43:38.951779 3874 raft_consensus.cc:686] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:39.075227 1668 rpcz_store.cc:276] 0806 03:43:37.360403 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.360581 (+ 178us) service_pool.cc:227] Handling call
0806 03:43:37.386593 (+ 26012us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:37.419770 (+ 33177us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:37.419800 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:37.465228 (+ 45428us) log.cc:842] Serialized 26 byte log entry
0806 03:43:38.387033 (+921805us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:38.387073 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.387138 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:39.074512 (+687374us) raft_consensus.cc:1671] finished
0806 03:43:39.074538 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:39.074701 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32748,"raft.queue_time_us":33277,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":37795,"thread_start_us":872508,"threads_started":1,"wal-append.queue_time_us":1426555}
W20260806 03:43:39.131306 727 connection.cc:232] RPC call timeout handler was delayed by -1.9047624247614294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260806 03:43:39.132889 727 connection.cc:232] RPC call timeout handler was delayed by -1.888562424806878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260806 03:43:39.133998 727 connection.cc:232] RPC call timeout handler was delayed by -1.8705624248832464s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1817s for a response
W20260806 03:43:39.134931 727 connection.cc:232] RPC call timeout handler was delayed by -1.8651624247431755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18230000000000002s for a response
W20260806 03:43:39.136013 727 connection.cc:232] RPC call timeout handler was delayed by -1.8633624247740954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
W20260806 03:43:39.137145 727 connection.cc:232] RPC call timeout handler was delayed by -1.8408624248113483s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
W20260806 03:43:39.138433 727 connection.cc:232] RPC call timeout handler was delayed by -1.8066624249331653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1888s for a response
W20260806 03:43:39.140030 727 connection.cc:232] RPC call timeout handler was delayed by -1.794062424916774s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
W20260806 03:43:39.141769 727 connection.cc:232] RPC call timeout handler was delayed by -1.7589624249376357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20260806 03:43:39.143100 727 connection.cc:232] RPC call timeout handler was delayed by -1.7391624248120934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
W20260806 03:43:39.144374 727 connection.cc:232] RPC call timeout handler was delayed by -1.7346624247729778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260806 03:43:39.145929 727 connection.cc:232] RPC call timeout handler was delayed by -1.724762424826622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260806 03:43:39.039590 4094 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260806 03:43:39.042634 4084 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:39.044639 4096 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
W20260806 03:43:39.045694 4097 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260806 03:43:39.045849 1670 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:39.046794 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2252) took 3494 ms. Trace:
I20260806 03:43:39.055286 3992 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:39.055713 773 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:39.037918 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 155678us
I20260806 03:43:38.946992 3876 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:38.753298 4060 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:38.957736 4085 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:38.935915 3972 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:39.502233 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:39.147739 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 297965us
W20260806 03:43:39.497505 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 696415us
W20260806 03:43:39.200965 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 408715us
I20260806 03:43:39.504390 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:39.504966 770 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:43:39.504966 616 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:39.497997 3955 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:39.321751 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:39.076779 1657 rpcz_store.cc:276] 0806 03:43:36.868684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.876815 (+ 8131us) service_pool.cc:227] Handling call
0806 03:43:36.998595 (+121780us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:36.998905 (+ 310us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:38.293025 (+1294120us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.867037 (+574012us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54004506d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:38.921740 (+ 54703us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1345351296}
W20260806 03:43:39.368701 4022 consensus_meta.cc:287] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.025s user 0.008s sys 0.005s
W20260806 03:43:39.376276 3960 consensus_meta.cc:287] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.833s user 0.008s sys 0.004s
W20260806 03:43:39.373559 4057 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.673s user 0.006s sys 0.004s
I20260806 03:43:39.161958 4079 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:39.105296 3450 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:39.496490 3609 consensus_meta.cc:287] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.788s user 0.007s sys 0.004s
I20260806 03:43:39.000706 3941 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:39.510803 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:39.348027 3905 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:39.512038 727 connection.cc:232] RPC call timeout handler was delayed by -1.7058624248020351s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
I20260806 03:43:39.085081 3980 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:39.636276 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:39.201402 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 411391us
I20260806 03:43:39.514014 4084 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:39.321383 4028 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.012s sys 0.008s
I20260806 03:43:39.514281 3972 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:39.000981 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 582) took 2130 ms. Trace:
I20260806 03:43:39.515687 3883 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:39.516741 3959 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.004s sys 0.004s
W20260806 03:43:39.333050 4100 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.399s user 0.000s sys 0.001s
I20260806 03:43:39.332080 3874 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:39.106768 3710 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:39.517699 4085 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:39.530681 770 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:39.688683 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:39.621212 4106 raft_consensus.cc:3032] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:39.518683 659 rpcz_store.cc:276] 0806 03:43:35.552102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.028484 (+1476382us) service_pool.cc:227] Handling call
0806 03:43:37.028926 (+ 442us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:38.388767 (+1359841us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:38.388796 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:38.447637 (+ 58841us) log.cc:842] Serialized 26 byte log entry
0806 03:43:38.519198 (+ 71561us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:38.519233 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.519363 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:39.046469 (+527106us) raft_consensus.cc:1671] finished
0806 03:43:39.046505 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:39.046720 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46034,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":13474,"thread_start_us":1373585,"threads_started":2,"wal-append.queue_time_us":100391}
I20260806 03:43:39.690513 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.539124 663 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:43:39.540158 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:39.539395 564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1811) took 2188 ms (client timeout 1988 ms). Trace:
I20260806 03:43:39.565140 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.596220 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.684820 614 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:39.692524 664 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 1.
I20260806 03:43:39.635699 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:39.661770 3959 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.806s user 0.004s sys 0.004s
I20260806 03:43:39.513266 776 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:39.631148 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.708635 546 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:39.631392 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.637509 661 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:39.691175 663 pending_rounds.cc:76] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Aborting all ops after (but not including) 2
I20260806 03:43:39.711174 663 pending_rounds.cc:97] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:43:39.711474 663 raft_consensus.cc:2866] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:39.537379 4022 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:39.638495 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.638985 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.634992 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:39.625912 727 connection.cc:232] RPC call timeout handler was delayed by -1.7049624249339104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
I20260806 03:43:39.565095 4024 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:39.593863 3933 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:39.592271 4057 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:39.592893 3862 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:39.640172 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 135376us
I20260806 03:43:39.801913 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.661978 771 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:43:39.594417 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:39.531147 3955 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.104s
I20260806 03:43:39.536972 1663 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
W20260806 03:43:39.639390 4028 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.928s user 0.014s sys 0.010s
W20260806 03:43:39.684036 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 143058us
I20260806 03:43:39.684370 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.689450 770 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:39.534261 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:39.535552 561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1791) took 2672 ms (client timeout 2001 ms). Trace:
I20260806 03:43:39.632550 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.638772 3609 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:39.625288 4109 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:39.691573 564 rpcz_store.cc:269] 0806 03:43:37.351033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.351205 (+ 172us) service_pool.cc:227] Handling call
0806 03:43:39.532303 (+2181098us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540031f6d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:39.539330 (+ 7027us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3111708544}
I20260806 03:43:39.696242 669 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:39.531733 3876 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:39.566061 3960 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:39.715165 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.773108 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:39.716353 727 connection.cc:232] RPC call timeout handler was delayed by -1.6941207819618285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260806 03:43:39.715229 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 188236us
I20260806 03:43:39.783077 668 raft_consensus.cc:2396] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:39.620867 3450 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:39.690061 3883 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:39.684528 542 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:39.621606 4060 raft_consensus.cc:686] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:39.774730 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.795141 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:39.795747 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 111015us
I20260806 03:43:39.692544 4106 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:39.795157 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:39.795347 674 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:39.641855 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:39.798568 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:39.594250 4079 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.083248 659 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:43:39.711799 3959 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:43:39.773866 3614 consensus_meta.cc:287] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.957s user 0.012s sys 0.000s
I20260806 03:43:39.642753 4111 consensus_queue.cc:1046] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20260806 03:43:39.902012 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:39.956483 499 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:39.930531 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:40.208833 544 raft_consensus.cc:2362] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5. Re-sending same reply.
I20260806 03:43:39.847021 558 raft_consensus.cc:2379] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:39.961370 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 366) took 1111 ms. Trace:
W20260806 03:43:39.956701 561 rpcz_store.cc:269] 0806 03:43:36.850645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.854455 (+ 3810us) service_pool.cc:227] Handling call
0806 03:43:39.522625 (+2668170us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b5400179654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:39.522919 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3934480640}
W20260806 03:43:39.967314 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 165380us
I20260806 03:43:39.844904 3862 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:39.976768 4109 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:39.994493 727 connection.cc:232] RPC call timeout handler was delayed by -1.6878624248784035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
I20260806 03:43:39.807014 3933 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:39.812796 4000 consensus_meta.cc:287] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.850s user 0.012s sys 0.000s
I20260806 03:43:39.995934 1671 raft_consensus.cc:2379] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:39.661695 1662 rpcz_store.cc:276] 0806 03:43:36.871015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.876734 (+ 5719us) service_pool.cc:227] Handling call
0806 03:43:36.940900 (+ 64166us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:37.457215 (+516315us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:37.458906 (+ 1691us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:37.489387 (+ 30481us) log.cc:842] Serialized 26 byte log entry
0806 03:43:38.281389 (+792002us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:38.281463 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:38.281576 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:38.952753 (+671177us) raft_consensus.cc:1671] finished
0806 03:43:38.954116 (+ 1363us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:38.999596 (+ 45480us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3451,"raft.queue_time_us":36148,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9617,"thread_start_us":1193317,"threads_started":2,"wal-append.queue_time_us":897651}
I20260806 03:43:39.885869 1657 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:39.998600 3960 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:39.999310 664 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:40.000375 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.000701 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.807732 4024 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.003405 1624 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:40.002823 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 288575us
I20260806 03:43:39.994777 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:39.685005 4101 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.223s
I20260806 03:43:40.015779 548 raft_consensus.cc:2354] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4.
I20260806 03:43:40.016703 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:39.902102 4113 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:40.016865 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:40.018332 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.019372 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.256867 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.257431 669 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:39.788036 4009 consensus_queue.cc:1046] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.218s
I20260806 03:43:40.022840 770 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:40.155709 769 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 03:43:40.157349 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.005511 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 365) took 1205 ms. Trace:
I20260806 03:43:40.259155 674 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:39.794915 4108 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.234s
I20260806 03:43:40.043560 3609 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:40.210181 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.267172 770 raft_consensus.cc:1261] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:40.211359 616 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:40.270115 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 58888.3us
I20260806 03:43:40.210071 1670 rpcz_store.cc:276] 0806 03:43:38.849543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.534029 (+684486us) service_pool.cc:227] Handling call
0806 03:43:39.961300 (+427271us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:40.274668 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 271372us
I20260806 03:43:40.279121 4114 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:40.279827 564 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:40.010332 4084 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.288745 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.258296 542 raft_consensus.cc:595] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:40.213871 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:40.157100 3614 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.214202 3357 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:43:40.223140 727 connection.cc:232] RPC call timeout handler was delayed by -1.6014624247327447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260806 03:43:40.324126 727 connection.cc:232] RPC call timeout handler was delayed by -1.5906624249182642s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
I20260806 03:43:40.229280 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:40.325255 727 connection.cc:232] RPC call timeout handler was delayed by -1.559162424877286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
W20260806 03:43:40.325588 727 connection.cc:232] RPC call timeout handler was delayed by -1.4682624249253422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2264s for a response
I20260806 03:43:40.325654 558 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:40.231463 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:40.011585 3883 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.160477 3980 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:40.281950 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:40.236491 4107 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.352s user 0.000s sys 0.000s
I20260806 03:43:40.236290 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:40.236840 4105 log.cc:924] Time spent T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.532s user 0.000s sys 0.001s
I20260806 03:43:40.238559 1657 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:40.244684 4060 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.239149 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.264134 1663 rpcz_store.cc:276] 0806 03:43:38.800009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.185601 (+385592us) service_pool.cc:227] Handling call
0806 03:43:39.969445 (+783844us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:40.241882 659 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:40.315470 4131 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.243587 3959 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.246672 671 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:40.247714 661 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:40.251034 3450 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:40.260193 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:40.262941 663 raft_consensus.cc:2379] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:40.212875 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 256472us
I20260806 03:43:40.369355 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.370036 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:40.295477 4129 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.300913 4049 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:40.305277 4007 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:40.307818 3917 raft_consensus.cc:2781] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:40.287019 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 744) took 1285 ms. Trace:
I20260806 03:43:40.310654 544 raft_consensus.cc:2379] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:40.313998 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:40.379541 1670 rpcz_store.cc:276] 0806 03:43:39.001440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.285686 (+1284246us) service_pool.cc:227] Handling call
0806 03:43:40.286077 (+ 391us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.286309 (+ 232us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:40.286514 (+ 205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:40.286570 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:40.286622 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.286690 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:40.286818 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:40.314433 664 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:40.381625 1670 raft_consensus.cc:1226] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:40.382411 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 589) took 1263 ms. Trace:
I20260806 03:43:40.382815 1670 rpcz_store.cc:276] 0806 03:43:39.119120 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.381019 (+1261899us) service_pool.cc:227] Handling call
0806 03:43:40.381398 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.382090 (+ 692us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.382314 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:40.315084 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 69255.8us
I20260806 03:43:40.326318 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.326668 561 raft_consensus.cc:2362] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5. Re-sending same reply.
I20260806 03:43:40.324990 674 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:43:40.235675 3940 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.173s user 0.011s sys 0.000s
I20260806 03:43:40.333411 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:40.231135 3876 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.347874 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 364) took 1773 ms. Trace:
I20260806 03:43:40.396885 1657 rpcz_store.cc:276] 0806 03:43:38.574872 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.237990 (+1663118us) service_pool.cc:227] Handling call
0806 03:43:40.238336 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.347604 (+109268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.347812 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:40.227129 3862 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:40.312368 542 raft_consensus.cc:603] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:40.231153 4000 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.243968 4109 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:40.354683 3357 raft_consensus.cc:686] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:40.356042 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.356554 563 raft_consensus.cc:2379] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 2: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:40.333233 4022 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:40.333441 4037 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:40.375504 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 115402us
I20260806 03:43:40.283447 4114 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.380335 770 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:40.336692 3980 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.382426 659 raft_consensus.cc:1261] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:40.393567 668 raft_consensus.cc:2396] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:43:40.371096 4125 log.cc:924] Time spent T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260806 03:43:40.355969 1658 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:40.433621 3357 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.280897 4106 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.435123 4106 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:40.438432 562 raft_consensus.cc:2354] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
W20260806 03:43:40.406591 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 81576us
I20260806 03:43:40.423048 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:40.423277 1624 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:40.419618 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:40.441324 1657 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:40.391605 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:40.396083 3940 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.371428 4131 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.407614 4049 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260806 03:43:40.400966 3609 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.389684 4129 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.413303 3814 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:40.475338 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.403491 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 742) took 1650 ms. Trace:
W20260806 03:43:40.654049 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 637219us
I20260806 03:43:40.549057 1657 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:40.655220 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:40.600656 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2164) took 3772 ms. Trace:
I20260806 03:43:40.469019 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.656531 659 raft_consensus.cc:2379] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:40.656337 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:40.656463 614 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:40.441844 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:40.657948 770 raft_consensus.cc:2379] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:40.474855 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.659137 674 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:40.536139 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:40.468199 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45017.2us
I20260806 03:43:40.468089 4037 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:40.412360 3917 raft_consensus.cc:686] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:40.424623 4009 consensus_queue.cc:1225] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer ada5ef6656cb4420baf77a4b4342909e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:43:40.614647 4114 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:40.425603 4079 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:40.660583 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:40.667740 4101 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:40.670727 3874 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260806 03:43:40.671813 4015 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:40.681524 4085 consensus_meta.cc:287] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.908s user 0.006s sys 0.003s
W20260806 03:43:40.673792 1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 734) took 2148 ms (client timeout 1560 ms). Trace:
W20260806 03:43:40.683045 1671 rpcz_store.cc:269] 0806 03:43:38.525398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.525570 (+ 172us) service_pool.cc:227] Handling call
0806 03:43:40.673550 (+2147980us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400529754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:40.673690 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2950035200}
I20260806 03:43:40.673084 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.669247 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.687736 770 raft_consensus.cc:2396] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:40.689540 669 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:40.674690 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 363) took 2379 ms. Trace:
I20260806 03:43:40.689785 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:40.548491 4133 log.cc:924] Time spent T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260806 03:43:40.551092 4024 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:40.567523 3960 raft_consensus.cc:3032] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:40.655879 772 rpcz_store.cc:276] 0806 03:43:36.827849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.966558 (+138709us) service_pool.cc:227] Handling call
0806 03:43:36.992666 (+ 26108us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:37.325110 (+332444us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:43:38.272980 (+947870us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.383894 (+110914us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:38.383978 (+ 84us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:38.384044 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.600372 (+2216328us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54001df954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:40.600423 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:40.600574 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":185906,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":95,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":53778,"spinlock_wait_cycles":783179264,"thread_start_us":1026020,"threads_started":2,"wal-append.queue_time_us":233011}
I20260806 03:43:40.690943 1664 rpcz_store.cc:276] 0806 03:43:38.295395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.314450 (+ 19055us) service_pool.cc:227] Handling call
0806 03:43:38.314853 (+ 403us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:38.649901 (+335048us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.674414 (+2024513us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b5400698554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:40.674631 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":662712064}
I20260806 03:43:40.660844 498 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:40.699128 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.699412 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:40.699718 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 308194us
I20260806 03:43:40.700385 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:40.727514 1657 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:40.701823 659 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:40.469288 3933 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:40.674899 4049 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:40.547746 4141 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.674043 3907 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:40.706548 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 50182.8us
I20260806 03:43:40.706630 3942 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:40.651849 3883 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:40.707325 4015 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:40.708640 3940 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:40.709513 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.659777 4152 raft_consensus.cc:2741] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:43:40.681957 4085 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.534652 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:40.711632 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:40.661880 4062 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.166s user 0.036s sys 0.063s
I20260806 03:43:40.566617 3614 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:40.718515 727 connection.cc:232] RPC call timeout handler was delayed by -1.3296624247450382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24180000000000001s for a response
I20260806 03:43:40.703567 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.731541 668 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:40.731949 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 71157.6us
W20260806 03:43:40.888298 727 connection.cc:232] RPC call timeout handler was delayed by -1.2072624247521162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2554s for a response
W20260806 03:43:40.888756 727 connection.cc:232] RPC call timeout handler was delayed by -1.1667624248657376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2599s for a response
I20260806 03:43:40.678488 4113 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:40.669956 4139 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:40.735213 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:40.894258 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 159139us
I20260806 03:43:40.894825 773 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:40.894815 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:40.735357 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:40.736075 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:40.736902 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:40.466064 3980 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:40.739517 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 351441us
I20260806 03:43:40.726824 3960 raft_consensus.cc:729] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:40.742038 674 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:40.672770 4037 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.729663 4079 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.738404 3907 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:40.733716 4024 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:40.671743 3917 consensus_queue.cc:237] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:40.738857 3942 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:40.870648 4166 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:40.696751 4029 consensus_queue.cc:1046] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20260806 03:43:40.744932 3933 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:40.876631 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.831979 615 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:40.678311 4101 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:40.833933 4028 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260806 03:43:40.887352 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.079694 770 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:40.889026 727 connection.cc:232] RPC call timeout handler was delayed by -1.0398624248337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.274s for a response
W20260806 03:43:41.080715 727 connection.cc:232] RPC call timeout handler was delayed by -1.0398624248337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.274s for a response
I20260806 03:43:40.888969 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:40.663966 1662 rpcz_store.cc:276] 0806 03:43:38.753317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.259056 (+1505739us) service_pool.cc:227] Handling call
0806 03:43:40.329067 (+ 70011us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.350485 (+ 21418us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:40.355817 (+ 5332us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:40.371250 (+ 15433us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:40.374959 (+ 3709us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.389494 (+ 14535us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:40.401673 (+ 12179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:40.832728 4015 raft_consensus.cc:686] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:40.831295 3883 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:41.085974 3933 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:40.896768 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.897655 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:40.886723 4062 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.396s user 0.036s sys 0.063s
W20260806 03:43:40.998339 4051 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 2.455s user 0.016s sys 0.061s
I20260806 03:43:40.742141 4141 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.001700 4131 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:40.964330 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.964955 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.960937 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.986611 766 raft_consensus.cc:2396] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:40.971971 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:40.891479 4113 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:40.898645 3940 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:40.899437 4114 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:40.998498 4159 log.cc:924] Time spent T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260806 03:43:40.964179 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:40.899812 4139 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.007380 548 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:41.097488 4060 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:41.075495 664 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:41.114928 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.032377 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.077463 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.117449 544 raft_consensus.cc:2354] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
W20260806 03:43:41.084393 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 195494us
W20260806 03:43:41.086571 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 254663us
I20260806 03:43:41.078847 3942 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.053960 4129 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:41.122601 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.090638 4146 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:41.090826 4015 consensus_queue.cc:237] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:41.091866 3883 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:41.034149 4007 consensus_meta.cc:287] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.638s user 0.027s sys 0.031s
I20260806 03:43:40.892329 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.136925 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:41.054721 4106 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.589s user 0.007s sys 0.004s
I20260806 03:43:41.046348 3651 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 03:43:41.055789 4163 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260806 03:43:41.098716 1671 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:41.089782 4166 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.100342 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:41.085182 4009 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:41.105559 614 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.094013 4024 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:41.109925 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.111162 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.112726 770 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:41.013197 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2177) took 2342 ms. Trace:
W20260806 03:43:41.149060 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 43608.1us
I20260806 03:43:41.149525 771 rpcz_store.cc:276] 0806 03:43:38.629272 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.704779 (+ 75507us) service_pool.cc:227] Handling call
0806 03:43:38.705167 (+ 388us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:39.810306 (+1105139us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:40.901384 (+1091078us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b5400070654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:40.971320 (+ 69936us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2158863616}
I20260806 03:43:41.149999 614 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.118947 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.151306 498 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.119215 615 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.007489 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.124924 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.126060 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.130865 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.133884 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.135586 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.039680 3960 consensus_queue.cc:260] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.139909 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.098027 661 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:41.118584 558 raft_consensus.cc:2354] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6.
I20260806 03:43:41.102151 4079 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.159235 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.158751 668 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.160310 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.160421 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.161124 544 raft_consensus.cc:3037] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.157663 614 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.171310 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.172078 669 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:41.173420 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.173928 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.176291 498 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.177652 498 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.179039 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:41.167886 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 48738.8us
I20260806 03:43:41.141780 3901 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:41.185372 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.185915 614 leader_election.cc:398] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
I20260806 03:43:41.186396 614 leader_election.cc:401] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:43:41.139847 4146 raft_consensus.cc:686] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:41.190814 4146 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.191636 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43080 (request call id 17) took 1561 ms. Trace:
I20260806 03:43:41.192363 542 rpcz_store.cc:276] 0806 03:43:39.629949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.649337 (+ 19388us) service_pool.cc:227] Handling call
0806 03:43:41.191251 (+1541914us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54002a0654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:41.191437 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402198784}
I20260806 03:43:41.195132 1657 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:43:41.151376 1662 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:41.183595 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.131181 3940 consensus_queue.cc:1046] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260806 03:43:41.182014 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:41.256378 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 74475.2us
I20260806 03:43:41.197530 664 raft_consensus.cc:2379] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:41.124296 4060 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:41.198472 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.197702 674 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
W20260806 03:43:41.221433 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 127596us
I20260806 03:43:41.241164 772 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:41.250118 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.179225 4085 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:41.170298 4062 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:41.241091 546 raft_consensus.cc:1226] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:41.181046 4106 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:41.249511 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.248972 563 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:41.259698 548 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:41.169476 3874 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:41.168743 3876 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:41.262701 3917 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5.
W20260806 03:43:41.263657 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 69649us
I20260806 03:43:41.213219 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.179553 4007 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:41.179155 3647 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:41.184593 4024 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:41.240363 544 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:41.287526 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:41.196763 3960 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.263952 4190 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:41.194648 4114 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:41.260780 3604 raft_consensus.cc:3032] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:41.265394 3959 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:41.164136 4009 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:41.200065 3901 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.333498 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 735) took 2807 ms. Trace:
I20260806 03:43:41.334501 542 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:43:41.333956 1660 rpcz_store.cc:276] 0806 03:43:38.525757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.529181 (+ 3424us) service_pool.cc:227] Handling call
0806 03:43:38.536000 (+ 6819us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:38.839905 (+303905us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:38.933219 (+ 93314us) log.cc:842] Serialized 14 byte log entry
0806 03:43:39.367227 (+434008us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:39.445873 (+ 78646us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:39.461423 (+ 15550us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.333182 (+1871759us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b54004f2754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:41.333246 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:41.333401 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":77936,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":9833,"spinlock_wait_cycles":616531200,"thread_start_us":267800,"threads_started":1,"wal-append.queue_time_us":840427}
I20260806 03:43:41.324968 3604 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:41.336575 3604 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.337869 3604 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:41.328219 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.304692 4101 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:41.250084 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:41.305197 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 133464us
I20260806 03:43:41.307926 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.311899 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.264329 3933 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:41.184386 4098 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20260806 03:43:41.348430 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.314527 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.316562 1657 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:41.320639 1671 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:41.323812 498 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.294260 4106 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.354672 4106 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:41.162770 4113 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.303395 3609 raft_consensus.cc:3032] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:41.300895 4062 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:41.391750 668 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:41.344129 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.409853 664 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:41.340727 771 raft_consensus.cc:2362] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4. Re-sending same reply.
I20260806 03:43:41.326557 4114 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:41.406199 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.412631 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1290) took 1027 ms. Trace:
I20260806 03:43:41.413128 771 rpcz_store.cc:276] 0806 03:43:40.385316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.319191 (+933875us) service_pool.cc:227] Handling call
0806 03:43:41.412538 (+ 93347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:41.416276 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.352383 661 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:41.365335 4176 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.251s user 0.000s sys 0.001s
W20260806 03:43:41.361920 4183 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260806 03:43:41.320333 4024 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.393765 3902 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:41.394593 4062 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.394976 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 738) took 2849 ms. Trace:
I20260806 03:43:41.432135 661 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:41.396575 4060 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:41.397130 3942 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.397718 3814 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.401052 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1293) took 1004 ms. Trace:
I20260806 03:43:41.438171 772 rpcz_store.cc:276] 0806 03:43:40.396686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.714978 (+318292us) service_pool.cc:227] Handling call
0806 03:43:41.400883 (+685905us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b5400321754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:41.400991 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":793856768}
I20260806 03:43:41.402637 3907 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.408860 1657 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:41.420647 664 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:41.404464 3609 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:41.342777 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.440785 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.441540 661 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
W20260806 03:43:41.443040 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 100396us
W20260806 03:43:41.403685 4177 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.290s user 0.002s sys 0.000s
I20260806 03:43:41.433439 668 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 03:43:41.410574 4180 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.118s user 0.002s sys 0.000s
I20260806 03:43:41.436036 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.403649 4131 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:43:41.444566 1622 leader_election.cc:334] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.832s (SENT)
I20260806 03:43:41.452826 771 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:41.453007 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.434711 4024 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.439004 1671 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:41.410578 4128 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 03:43:41.421558 4114 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.430745 3613 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:41.453715 772 raft_consensus.cc:2362] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6. Re-sending same reply.
I20260806 03:43:41.431391 4139 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.444532 614 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.454603 1624 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:41.436604 3902 consensus_queue.cc:1225] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 03:43:41.453101 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.439675 3814 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:41.452299 4057 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.363s user 0.008s sys 0.022s
I20260806 03:43:41.463363 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:41.467756 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.464859 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.469693 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.470328 772 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:41.471073 614 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.469928 771 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:41.473752 616 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.469866 1671 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:41.452525 4060 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:41.475943 614 leader_election.cc:398] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6.
I20260806 03:43:41.478907 614 leader_election.cc:401] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260806 03:43:41.467964 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 55919us
I20260806 03:43:41.476199 728 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:41.476637 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 70535us
I20260806 03:43:41.439709 3960 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 03:43:41.459259 3613 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.467029 4101 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:41.488569 3613 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:41.493994 4182 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.294s user 0.000s sys 0.002s
I20260806 03:43:41.484308 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:41.470441 4057 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.387s user 0.009s sys 0.022s
I20260806 03:43:41.487849 3960 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:41.467523 4141 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:41.462054 4079 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.473160 3933 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:43:41.496287 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 412707us
I20260806 03:43:41.446182 3874 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260806 03:43:41.462275 4191 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:41.474742 4142 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:41.469192 4131 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:41.434226 1658 rpcz_store.cc:276] 0806 03:43:38.545599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.834861 (+289262us) service_pool.cc:227] Handling call
0806 03:43:38.934277 (+ 99416us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.434607 (+1500330us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.394717 (+960110us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540069d7d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:41.394911 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3220224256}
I20260806 03:43:41.510454 3960 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:41.541724 4200 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:41.520840 4192 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:41.542301 4200 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.495741 4007 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:41.543443 4007 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:41.544003 4007 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:41.544588 4200 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:41.521644 4178 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.296s user 0.000s sys 0.002s
I20260806 03:43:41.547989 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 740) took 2983 ms. Trace:
I20260806 03:43:41.548059 4114 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.516916 3917 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6.
I20260806 03:43:41.495098 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.517879 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.485628 4193 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:41.485146 4139 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:41.472884 3614 consensus_queue.cc:1046] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 03:43:41.504082 1660 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:41.558929 4141 raft_consensus.cc:3032] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:41.559293 4141 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:41.535332 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.560026 771 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.548599 1665 rpcz_store.cc:276] 0806 03:43:38.564519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.788189 (+1223670us) service_pool.cc:227] Handling call
0806 03:43:39.968762 (+180573us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:40.731116 (+762354us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:40.734118 (+ 3002us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:40.885486 (+151368us) log.cc:842] Serialized 26 byte log entry
0806 03:43:41.036382 (+150896us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:41.040741 (+ 4359us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.076913 (+ 36172us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:41.547694 (+470781us) raft_consensus.cc:1671] finished
0806 03:43:41.547731 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:41.547909 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":182422,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":14588,"thread_start_us":596258,"threads_started":2,"wal-append.queue_time_us":217292}
I20260806 03:43:41.560209 4141 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.485879 4166 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:41.526651 3933 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:41.539803 727 leader_election.cc:334] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.675s (SENT)
I20260806 03:43:41.562942 4141 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:41.563432 4139 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.558140 766 raft_consensus.cc:2396] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:41.563854 4139 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.564702 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.565274 766 raft_consensus.cc:2362] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5. Re-sending same reply.
I20260806 03:43:41.485608 3647 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260806 03:43:41.562909 3933 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:41.566983 3942 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.481940 3959 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260806 03:43:41.553731 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:41.548856 3883 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.566110 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.574294 4193 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:41.592319 3814 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.541714 3960 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.578498 3980 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:41.609556 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 128947us
I20260806 03:43:41.579484 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:41.582979 772 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:41.538650 4009 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:41.586201 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 120053us
I20260806 03:43:41.615864 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.579533 4114 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:41.619701 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.590502 771 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:41.480801 3609 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.591566 766 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:41.592803 4193 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.594750 3907 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.598527 3613 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.575134 4079 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 03:43:41.603304 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 49626.2us
W20260806 03:43:41.612269 727 leader_election.cc:334] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.850s (SENT)
I20260806 03:43:41.612661 661 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:41.633543 727 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.614856 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1294) took 1214 ms. Trace:
I20260806 03:43:41.634291 772 rpcz_store.cc:276] 0806 03:43:40.400530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.457777 (+1057247us) service_pool.cc:227] Handling call
0806 03:43:41.614795 (+157018us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:41.621076 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:41.607467 3647 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:41.647570 3647 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:41.648149 3647 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.619581 1665 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.593914 4131 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:41.643491 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:41.650455 1657 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44d06aad0812478bb2270a2c80917306"
dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:55938
I20260806 03:43:41.650776 1668 raft_consensus.cc:2362] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5. Re-sending same reply.
W20260806 03:43:41.651336 1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 743) took 2698 ms (client timeout 1666 ms). Trace:
I20260806 03:43:41.629393 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:41.651685 1668 rpcz_store.cc:269] 0806 03:43:38.953413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.187534 (+234121us) service_pool.cc:227] Handling call
0806 03:43:41.643341 (+2455807us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:41.651263 (+ 7922us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134594944}
I20260806 03:43:41.640295 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.631821 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.653070 770 raft_consensus.cc:2379] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:41.604597 4062 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.637918 546 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:41.653129 668 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:43:41.638512 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.657730 561 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:41.658015 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.638408 3933 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.640275 3613 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:41.631247 3883 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:41.623245 3959 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260806 03:43:41.629120 3960 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.641783 3609 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.642237 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:41.642765 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.643445 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.644130 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.644964 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:41.645407 727 leader_election.cc:334] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.941s (SENT)
I20260806 03:43:41.645581 3980 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:41.645911 4128 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:41.647606 4106 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.620900 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.659168 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.636394 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.774290 661 raft_consensus.cc:2396] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:41.663563 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:41.719993 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 62076.6us
I20260806 03:43:41.776067 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:41.628345 4166 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:41.779927 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.659526 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 479) took 1364 ms. Trace:
I20260806 03:43:41.780835 770 rpcz_store.cc:276] 0806 03:43:40.288631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.640029 (+1351398us) service_pool.cc:227] Handling call
0806 03:43:41.653349 (+ 13320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:41.677652 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2185) took 2166 ms. Trace:
I20260806 03:43:41.783138 727 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.666102 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.783212 1624 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.719422 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.784230 546 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:41.675812 562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 318) took 2043 ms (client timeout 2612 ms). Trace:
I20260806 03:43:41.784916 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.782071 773 rpcz_store.cc:276] 0806 03:43:39.511671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.512793 (+ 1122us) service_pool.cc:227] Handling call
0806 03:43:39.513094 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.897558 (+1384464us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.677309 (+779751us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001031d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:41.677572 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3177302912}
I20260806 03:43:41.786356 661 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:41.782092 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.788080 770 raft_consensus.cc:2396] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:41.638159 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 745) took 2579 ms. Trace:
I20260806 03:43:41.790302 772 raft_consensus.cc:2396] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:41.786131 771 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
W20260806 03:43:41.785487 562 rpcz_store.cc:269] 0806 03:43:39.632870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.238876 (+606006us) service_pool.cc:227] Handling call
0806 03:43:41.675580 (+1436704us) spinlock_profiling.cc:243] Waited 87.1 ms on lock 0x7b540030e654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:41.675744 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200407808}
I20260806 03:43:41.794229 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:41.797649 669 raft_consensus.cc:2379] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:41.799074 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.799093 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.800570 668 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:41.800768 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.805522 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.806128 1671 raft_consensus.cc:1226] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:41.768826 3332 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:41.808360 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.809569 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.808737 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.808398 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:41.811077 1622 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:41.807888 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.812188 773 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:41.814280 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.844079 771 raft_consensus.cc:2396] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:41.804402 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.784078 3883 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.830152 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:41.845235 772 raft_consensus.cc:2396] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:41.845381 661 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:41.812798 3614 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260806 03:43:41.831789 4000 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.832405 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.845679 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.833475 4106 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:41.834854 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:41.658739 4009 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:41.798164 1663 rpcz_store.cc:276] 0806 03:43:39.058543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.388831 (+1330288us) service_pool.cc:227] Handling call
0806 03:43:40.416324 (+ 27493us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.543974 (+127650us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:40.545547 (+ 1573us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:40.548236 (+ 2689us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:40.548288 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.637907 (+1089619us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54004f4cd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:41.637953 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:41.638092 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":253778816}
I20260806 03:43:41.838742 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2169) took 4532 ms. Trace:
I20260806 03:43:41.839310 3980 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:41.789512 4142 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:43:41.840739 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 46409.1us
I20260806 03:43:41.842245 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:41.842866 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:41.814394 770 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:41.846458 546 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:41.659312 4146 consensus_queue.cc:1046] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.196s
I20260806 03:43:41.842823 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:42.244838 4221 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260806 03:43:41.867020 4101 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:42.245302 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:41.846607 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:42.246127 773 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:41.857995 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:41.817433 3332 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:42.246793 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:41.860127 1657 raft_consensus.cc:595] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:42.263283 1657 raft_consensus.cc:603] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:41.856323 558 raft_consensus.cc:2354] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:41.805753 4114 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:43:42.248270 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2193) took 1608 ms. Trace:
I20260806 03:43:42.252789 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:42.252971 1624 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:42.264344 770 rpcz_store.cc:276] 0806 03:43:40.639606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.807496 (+1167890us) service_pool.cc:227] Handling call
0806 03:43:42.248212 (+440716us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:42.267629 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 409739us
I20260806 03:43:41.959263 544 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:42.274185 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 475) took 1997 ms. Trace:
I20260806 03:43:42.274739 770 rpcz_store.cc:276] 0806 03:43:40.276787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.273105 (+1996318us) service_pool.cc:227] Handling call
0806 03:43:42.273526 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.273755 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.273791 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.273850 (+ 59us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:42.273903 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.273972 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.274112 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.255203 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:42.277055 770 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:42.278803 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1303) took 1223 ms. Trace:
W20260806 03:43:42.189347 4223 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.349s user 0.001s sys 0.000s
I20260806 03:43:42.264398 661 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:42.237211 4087 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 3.701s user 0.011s sys 0.017s
I20260806 03:43:42.301819 558 raft_consensus.cc:1226] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:42.301858 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2190) took 2033 ms. Trace:
I20260806 03:43:42.253175 775 rpcz_store.cc:276] 0806 03:43:37.305913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.423658 (+117745us) service_pool.cc:227] Handling call
0806 03:43:37.461720 (+ 38062us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:38.482841 (+1021121us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:38.504424 (+ 21583us) log.cc:842] Serialized 14 byte log entry
0806 03:43:38.609023 (+104599us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:38.609777 (+ 754us) log.cc:842] Serialized 52 byte log entry
0806 03:43:38.609940 (+ 163us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:38.610076 (+ 136us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:41.838255 (+3228179us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5400185954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:41.838310 (+ 55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:41.838383 (+ 73us) raft_consensus.cc:1671] finished
0806 03:43:41.838484 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:41.838656 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10411,"prepare.queue_time_us":21259,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":3698,"raft.queue_time_us":19215,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":19598,"spinlock_wait_cycles":1881481472,"thread_start_us":293586,"threads_started":2,"wal-append.queue_time_us":195820,"wal-append.run_cpu_time_us":16685,"wal-append.run_wall_time_us":1821388}
I20260806 03:43:42.302795 772 rpcz_store.cc:276] 0806 03:43:40.268999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.271414 (+2002415us) service_pool.cc:227] Handling call
0806 03:43:42.301377 (+ 29963us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.301554 (+ 177us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.301578 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.301616 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:42.301651 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.301697 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.301799 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:42.304178 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.003s sys 0.007s
I20260806 03:43:41.814977 4037 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:42.051780 3814 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:42.268074 3332 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:42.271027 4106 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:41.901187 668 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:43:42.243873 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 598493us
W20260806 03:43:42.319206 727 leader_election.cc:334] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.963s (SENT)
I20260806 03:43:42.319823 727 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:42.188551 4220 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20260806 03:43:42.321805 3980 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:42.267684 4114 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:42.300206 4141 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:42.299512 771 rpcz_store.cc:276] 0806 03:43:41.055219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.856882 (+801663us) service_pool.cc:227] Handling call
0806 03:43:42.278705 (+421823us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.249356 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:42.254069 4139 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.690s user 0.011s sys 0.000s
I20260806 03:43:42.269660 4146 consensus_queue.cc:1225] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer ada5ef6656cb4420baf77a4b4342909e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:43:41.962222 3917 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:42.309900 773 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:42.067608 4060 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:42.292151 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 476) took 2015 ms. Trace:
I20260806 03:43:42.314308 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:42.313871 3933 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.278488 4142 raft_consensus.cc:686] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:42.281472 4101 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:42.359881 4146 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:42.360133 4101 raft_consensus.cc:686] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:42.360571 4146 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.361088 4101 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:41.877193 4057 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.454s
I20260806 03:43:42.363695 4057 raft_consensus.cc:3032] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:43:42.364210 4057 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:42.331459 4141 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:42.364923 4141 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:42.326301 775 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 2. Current term is 3. Ops: []
I20260806 03:43:41.997641 4007 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.311151 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:42.352540 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1289) took 1967 ms. Trace:
I20260806 03:43:42.355890 1668 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:42.280876 4009 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:42.357203 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:42.359062 4142 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:42.366781 1664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.050s sys 0.088s
W20260806 03:43:42.390909 1664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s user 0.053s sys 0.091s
W20260806 03:43:42.331228 4022 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.013s sys 0.020s
I20260806 03:43:42.329855 4114 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:42.328547 4106 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:42.335922 4024 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.869s user 0.009s sys 0.003s
W20260806 03:43:42.392705 4022 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.014s sys 0.020s
I20260806 03:43:42.406945 4009 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:42.408367 3647 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.760s user 0.009s sys 0.000s
I20260806 03:43:42.408836 3647 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.393082 3613 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:42.411384 4146 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.365461 4057 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.412582 4057 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:43:42.412707 1665 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.764s user 0.009s sys 0.000s
I20260806 03:43:42.413149 1665 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:42.423727 4106 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.424379 4050 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:42.396287 4024 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:42.408370 3960 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.630s user 0.009s sys 0.000s
I20260806 03:43:42.355614 3917 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.355470 4139 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.373834 4060 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.440517 4114 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.443341 4114 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:42.433167 4141 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.448904 4141 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.445813 4139 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:42.494424 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:42.576380 767 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 474) took 3787 ms (client timeout 1913 ms). Trace:
I20260806 03:43:42.500366 561 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260806 03:43:42.577194 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 83348.7us
I20260806 03:43:42.501894 564 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:42.304560 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.515s user 0.003s sys 0.007s
I20260806 03:43:42.579351 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1186) took 1573 ms. Trace:
I20260806 03:43:42.579810 564 rpcz_store.cc:276] 0806 03:43:41.006414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.168712 (+162298us) service_pool.cc:227] Handling call
0806 03:43:42.579019 (+1410307us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540011e9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.579245 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2744768896}
I20260806 03:43:42.313015 661 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
W20260806 03:43:42.582211 776 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2167) took 5276 ms (client timeout 3050 ms). Trace:
W20260806 03:43:42.582830 776 rpcz_store.cc:269] 0806 03:43:37.305321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:37.361813 (+ 56492us) service_pool.cc:227] Handling call
0806 03:43:42.581637 (+5219824us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b5400311354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.581947 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4739312896}
I20260806 03:43:42.506552 668 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 03:43:42.512432 766 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:42.386750 771 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:42.368472 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2191) took 2001 ms. Trace:
I20260806 03:43:42.513680 770 rpcz_store.cc:276] 0806 03:43:40.277217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.276418 (+1999201us) service_pool.cc:227] Handling call
0806 03:43:42.276805 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.291735 (+ 14930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.292062 (+ 327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.380543 773 rpcz_store.cc:276] 0806 03:43:40.384923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.351728 (+1966805us) service_pool.cc:227] Handling call
0806 03:43:42.352060 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.352236 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.352264 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.352304 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:42.352337 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.352389 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.352483 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:42.519170 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.000s sys 0.010s
I20260806 03:43:42.521334 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:42.523075 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:42.535597 499 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:42.538132 3647 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:42.519160 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 477) took 2235 ms. Trace:
I20260806 03:43:42.539382 3334 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:42.544845 3907 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:42.542235 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:42.551267 3917 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:42.557634 547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1797) took 5704 ms (client timeout 1547 ms). Trace:
I20260806 03:43:42.430866 4007 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.559772 4010 raft_consensus.cc:3032] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:43:42.567540 659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 296) took 2083 ms (client timeout 1708 ms). Trace:
W20260806 03:43:42.569473 3609 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.791s user 0.009s sys 0.013s
W20260806 03:43:42.574394 665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2269) took 2977 ms (client timeout 1796 ms). Trace:
I20260806 03:43:42.576750 4000 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:42.694669 3334 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 03:43:42.665977 4254 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.209s user 0.001s sys 0.000s
I20260806 03:43:42.672468 1668 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:42.697193 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:42.698206 1668 raft_consensus.cc:2379] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:42.586699 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:42.587235 4129 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:43:42.604010 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 282607us
W20260806 03:43:42.576951 767 rpcz_store.cc:269] 0806 03:43:38.789139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:38.797192 (+ 8053us) service_pool.cc:227] Handling call
0806 03:43:42.575799 (+3778607us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b540030ff54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.576096 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5133224832}
I20260806 03:43:42.615149 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2286) took 2237 ms. Trace:
I20260806 03:43:42.615784 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:42.618969 768 rpcz_store.cc:276] 0806 03:43:40.367246 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.348923 (+1981677us) service_pool.cc:227] Handling call
0806 03:43:42.367798 (+ 18875us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.368040 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.368075 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.368132 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:42.368183 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.368249 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.368377 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.619619 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 753) took 1138 ms. Trace:
W20260806 03:43:42.629519 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.000s sys 0.011s
I20260806 03:43:42.638525 1670 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9. Ops: []
I20260806 03:43:42.662073 775 rpcz_store.cc:276] 0806 03:43:40.283037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.303685 (+2020648us) service_pool.cc:227] Handling call
0806 03:43:42.318037 (+ 14352us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.387322 (+ 69285us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.518631 (+131309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.665926 4141 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:42.667815 1658 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:42.672465 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 481) took 2274 ms. Trace:
I20260806 03:43:42.672367 4101 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:42.664283 547 rpcz_store.cc:269] 0806 03:43:36.852661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:36.852815 (+ 154us) service_pool.cc:227] Handling call
0806 03:43:42.557045 (+5704230us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b540011eed4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.557348 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4828785536}
W20260806 03:43:42.688755 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 153679us
I20260806 03:43:42.663528 3907 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:42.718415 773 rpcz_store.cc:276] 0806 03:43:40.396301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.668914 (+2272613us) service_pool.cc:227] Handling call
0806 03:43:42.669619 (+ 705us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.670113 (+ 494us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.670153 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.670245 (+ 92us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:42.670358 (+ 113us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.670436 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.670606 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:42.690613 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 148915us
I20260806 03:43:42.691738 3917 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:42.673439 4010 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:42.673709 659 rpcz_store.cc:269] 0806 03:43:40.484150 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.689476 (+205326us) service_pool.cc:227] Handling call
0806 03:43:42.567014 (+1877538us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b54002ce054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.567316 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21045504}
W20260806 03:43:42.674223 665 rpcz_store.cc:269] 0806 03:43:39.596631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.617983 (+ 21352us) service_pool.cc:227] Handling call
0806 03:43:42.573903 (+2955920us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b540015a9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.574145 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2210487680}
I20260806 03:43:42.585939 4024 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.729640 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2192) took 2317 ms. Trace:
I20260806 03:43:42.709486 1671 rpcz_store.cc:276] 0806 03:43:41.480926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.481189 (+ 263us) service_pool.cc:227] Handling call
0806 03:43:41.488815 (+ 7626us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:41.806508 (+317693us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.619086 (+812578us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400620554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:42.619418 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729320832}
I20260806 03:43:42.836671 776 raft_consensus.cc:2396] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:42.800202 770 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:43:42.707015 663 rpcz_store.cc:276] 0806 03:43:40.377898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.395333 (+ 17435us) service_pool.cc:227] Handling call
0806 03:43:40.395692 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:40.431310 (+ 35618us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:41.645447 (+1214137us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b1800171c40. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f321336e4ca 00007f321ca4ff99 00007f3213232990 00007f32133f4864 00007f32133f4155 00007f3219cf4167 00007f3216329807 00007f3216338f0f 00007f3216338e88
0806 03:43:41.645717 (+ 270us) log.cc:842] Serialized 14 byte log entry
0806 03:43:41.784783 (+139066us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:42.570297 (+785514us) spinlock_profiling.cc:243] Waited 5.52 ms on lock 0x7b2c00000238. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:43:42.571473 (+ 1176us) log.cc:842] Serialized 26 byte log entry
0806 03:43:42.571871 (+ 398us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:42.572640 (+ 769us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.573060 (+ 420us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.573271 (+ 211us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:42.614627 (+ 41356us) raft_consensus.cc:1671] finished
0806 03:43:42.614655 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.614933 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46889,"prepare.queue_time_us":1210431,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":106,"raft.queue_time_us":94325,"raft.run_cpu_time_us":767,"raft.run_wall_time_us":54318,"spinlock_wait_cycles":70061696,"thread_start_us":138611,"threads_started":1,"wal-append.queue_time_us":405168}
I20260806 03:43:42.730333 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 748) took 2356 ms. Trace:
I20260806 03:43:42.793725 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:42.839473 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 210026us
I20260806 03:43:42.840132 767 rpcz_store.cc:276] 0806 03:43:40.411917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.717140 (+2305223us) service_pool.cc:227] Handling call
0806 03:43:42.717753 (+ 613us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.728570 (+ 10817us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.728620 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.728709 (+ 89us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:42.728753 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.728826 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.728948 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.841513 1670 rpcz_store.cc:276] 0806 03:43:40.371821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.396856 (+ 25035us) service_pool.cc:227] Handling call
0806 03:43:42.613552 (+2216696us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:42.614040 (+ 488us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.710028 (+ 95988us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.728028 (+ 18000us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6933376}
I20260806 03:43:42.849789 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:42.850790 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 348) took 1061 ms. Trace:
I20260806 03:43:42.851234 544 rpcz_store.cc:276] 0806 03:43:41.788969 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.842000 (+ 53031us) service_pool.cc:227] Handling call
0806 03:43:42.850287 (+1008287us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540032b4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.850548 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259121920}
I20260806 03:43:42.854327 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2180) took 3355 ms. Trace:
I20260806 03:43:42.854846 769 rpcz_store.cc:276] 0806 03:43:39.498351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.498599 (+ 248us) service_pool.cc:227] Handling call
0806 03:43:39.498974 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:40.258825 (+759851us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.853748 (+2594923us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b5400186354. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:42.854110 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1345212672}
I20260806 03:43:42.856199 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 482) took 2193 ms. Trace:
I20260806 03:43:42.856684 776 rpcz_store.cc:276] 0806 03:43:40.662329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.854912 (+2192583us) service_pool.cc:227] Handling call
0806 03:43:42.855397 (+ 485us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.855797 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.855829 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.855880 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:42.855932 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.855993 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.856129 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.808458 3332 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.861091 3332 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.815349 3902 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:42.862498 3902 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.864902 3902 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.814316 3907 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.792721 3917 raft_consensus.cc:686] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:42.814615 4141 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:42.799868 727 leader_election.cc:334] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.001s (SENT)
I20260806 03:43:42.868731 768 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:43:42.887355 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 484) took 2206 ms. Trace:
I20260806 03:43:42.891307 769 rpcz_store.cc:276] 0806 03:43:40.674031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.855838 (+2181807us) service_pool.cc:227] Handling call
0806 03:43:42.856302 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.856602 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.856632 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.856689 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:42.856720 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.856767 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.880619 (+ 23852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.899094 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1297) took 2330 ms. Trace:
I20260806 03:43:42.899572 768 rpcz_store.cc:276] 0806 03:43:40.563548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.787703 (+2224155us) service_pool.cc:227] Handling call
0806 03:43:42.868324 (+ 80621us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:42.893818 (+ 25494us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.894101 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.868036 3907 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:42.903872 1670 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260806 03:43:42.909008 501 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:42.910197 4024 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:42.910878 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:42.911849 4024 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:42.913161 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:42.913237 4165 consensus_queue.cc:1046] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:42.915072 4095 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:42.915498 4095 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:42.916059 4095 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:42.916376 4024 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:42.916652 4057 raft_consensus.cc:3032] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:42.917320 4057 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:42.917716 4024 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.915468 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:42.919489 3334 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:42.925755 3883 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:42.926108 4158 log.cc:924] Time spent T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 2.040s user 0.006s sys 0.000s
I20260806 03:43:42.927017 1660 raft_consensus.cc:2396] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:42.927529 769 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:43:42.928314 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 487) took 1600 ms. Trace:
I20260806 03:43:42.928692 769 rpcz_store.cc:276] 0806 03:43:41.327678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.898075 (+1570397us) service_pool.cc:227] Handling call
0806 03:43:42.927129 (+ 29054us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.928002 (+ 873us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.928232 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:42.930552 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 100361us
I20260806 03:43:42.931475 1670 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:42.931941 1622 leader_election.cc:334] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.122s (SENT)
I20260806 03:43:42.932341 771 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:42.932657 3651 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:42.933393 3651 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.933836 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:42.934158 771 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:42.937346 4060 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:42.937107 4010 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.938550 4010 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:42.866360 4113 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:42.939774 3651 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.933414 4048 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:42.941143 4048 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:42.943147 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 143319us
I20260806 03:43:42.943902 4048 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.944669 4048 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election won for term 4
W20260806 03:43:42.945140 727 leader_election.cc:334] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.128s (SENT)
I20260806 03:43:42.949716 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2295) took 1324 ms. Trace:
I20260806 03:43:42.798534 4101 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.950145 665 rpcz_store.cc:276] 0806 03:43:41.624926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.896258 (+1271332us) service_pool.cc:227] Handling call
0806 03:43:42.948972 (+ 52714us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.949338 (+ 366us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.949364 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.949425 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:42.949462 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.949523 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.949632 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.952625 4101 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.870214 3917 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:42.963578 771 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:42.962415 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1307) took 1392 ms. Trace:
I20260806 03:43:42.966147 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1302) took 2088 ms. Trace:
I20260806 03:43:42.966768 776 rpcz_store.cc:276] 0806 03:43:40.877999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.857845 (+1979846us) service_pool.cc:227] Handling call
0806 03:43:42.858359 (+ 514us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.926704 (+ 68345us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.963403 (+ 36699us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.965854 (+ 2451us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:42.965909 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.965966 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.966093 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.967974 769 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:42.967913 770 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:42.965996 766 rpcz_store.cc:276] 0806 03:43:41.569832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.644678 (+ 74846us) service_pool.cc:227] Handling call
0806 03:43:42.962123 (+1317445us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b5400321754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:42.962306 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1686531840}
I20260806 03:43:42.956919 4141 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:42.981787 663 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:42.983676 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 45544.2us
I20260806 03:43:42.945168 4048 raft_consensus.cc:686] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:42.983371 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2296) took 1358 ms. Trace:
I20260806 03:43:42.962766 4255 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:42.985986 663 rpcz_store.cc:276] 0806 03:43:41.625188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.870810 (+1245622us) service_pool.cc:227] Handling call
0806 03:43:42.920390 (+ 49580us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.983044 (+ 62654us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.983278 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:42.961974 4101 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260806 03:43:42.976190 3334 raft_consensus.cc:686] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:42.961143 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:42.970510 4057 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:43.011713 727 leader_election.cc:334] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.554s (SENT)
I20260806 03:43:43.011265 3334 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.012308 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:43.014075 4050 consensus_meta.cc:287] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.589s user 0.014s sys 0.002s
I20260806 03:43:43.015681 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2297) took 1390 ms. Trace:
I20260806 03:43:43.016098 659 rpcz_store.cc:276] 0806 03:43:41.625444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.926582 (+1301138us) service_pool.cc:227] Handling call
0806 03:43:42.994336 (+ 67754us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.015205 (+ 20869us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:43.015232 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:43.015405 (+ 173us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:43.015451 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.015512 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.015622 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.017025 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.017639 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 593) took 1582 ms. Trace:
I20260806 03:43:43.018029 1662 rpcz_store.cc:276] 0806 03:43:41.435059 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.437313 (+ 2254us) service_pool.cc:227] Handling call
0806 03:43:42.960785 (+1523472us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:43.017545 (+ 56760us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9725312}
I20260806 03:43:43.019264 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:43.020143 669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 320) took 2007 ms (client timeout 1913 ms). Trace:
W20260806 03:43:43.020586 669 rpcz_store.cc:269] 0806 03:43:41.012404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.172986 (+160582us) service_pool.cc:227] Handling call
0806 03:43:43.019687 (+1846701us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54000654d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.019917 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1433095552}
I20260806 03:43:43.022830 667 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:43.021953 4095 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.025138 4095 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.025193 4101 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:43.025648 4101 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:43.026628 4130 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.026717 4101 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.027227 4130 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:43.028465 4101 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.033032 4164 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.035617 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 478) took 2714 ms. Trace:
I20260806 03:43:43.036096 772 rpcz_store.cc:276] 0806 03:43:40.288248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.304025 (+2015777us) service_pool.cc:227] Handling call
0806 03:43:42.313588 (+ 9563us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.313840 (+ 252us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:43:42.608564 (+294724us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.978686 (+370122us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.986185 (+ 7499us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:42.993296 (+ 7111us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.998331 (+ 5035us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.002280 (+ 3949us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":300258,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":4708,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3344,"thread_start_us":381353,"threads_started":2}
I20260806 03:43:43.037947 1622 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:43.004359 4108 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.040977 4108 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.042968 4141 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.043556 4108 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.006348 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2290) took 1343 ms. Trace:
I20260806 03:43:43.044266 661 rpcz_store.cc:276] 0806 03:43:41.621882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.583501 (+961619us) service_pool.cc:227] Handling call
0806 03:43:42.584125 (+ 624us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.586090 (+ 1965us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:43:42.622523 (+ 36433us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.962803 (+340280us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.963151 (+ 348us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.963282 (+ 131us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:42.963329 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.963393 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.965506 (+ 2113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35252,"prepare.queue_time_us":41230,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":91,"raft.queue_time_us":75107,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":9028,"spinlock_wait_cycles":14286976,"thread_start_us":185134,"threads_started":2,"wal-append.queue_time_us":301769}
I20260806 03:43:43.004602 4000 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.042s
I20260806 03:43:43.046795 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:42.986025 3883 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:43.007078 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 91747.3us
I20260806 03:43:43.048247 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.048442 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.008894 4255 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:42.987430 3651 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:43.050870 4255 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.050817 3651 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.052232 4210 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.010102 771 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:43:43.053369 3651 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:43.012410 3902 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:43.011963 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2300) took 1385 ms. Trace:
I20260806 03:43:43.054075 663 rpcz_store.cc:276] 0806 03:43:41.626235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.995758 (+1369523us) service_pool.cc:227] Handling call
0806 03:43:43.011132 (+ 15374us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.011549 (+ 417us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:43.011577 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:43.011628 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:43.011676 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.011745 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.011881 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.055624 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:43.055934 4263 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.117s user 0.002s sys 0.001s
I20260806 03:43:43.056869 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:43.050279 501 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.058753 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 761) took 1244 ms. Trace:
I20260806 03:43:43.058979 3883 consensus_queue.cc:1046] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:43:43.060325 3883 raft_consensus.cc:3032] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:43.060685 3883 raft_consensus.cc:729] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:43.060940 4122 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:43.062211 663 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:43.058761 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 57703.2us
W20260806 03:43:43.064342 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 58667.1us
I20260806 03:43:43.065680 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55938 (request call id 1) took 1417 ms. Trace:
I20260806 03:43:43.066118 1657 rpcz_store.cc:276] 0806 03:43:41.647802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.650268 (+ 2466us) service_pool.cc:227] Handling call
0806 03:43:43.065260 (+1414992us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54004af954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.065487 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":480572672}
I20260806 03:43:43.067247 4191 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.068174 1658 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:43.067907 4191 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.068820 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 760) took 1259 ms. Trace:
I20260806 03:43:43.069185 1658 rpcz_store.cc:276] 0806 03:43:41.809303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.397225 (+587922us) service_pool.cc:227] Handling call
0806 03:43:43.068716 (+671491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.059474 1663 rpcz_store.cc:276] 0806 03:43:41.813736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.057352 (+1243616us) service_pool.cc:227] Handling call
0806 03:43:43.057799 (+ 447us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.058082 (+ 283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:43.058126 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:43.058204 (+ 78us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:43.058265 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.058342 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.058546 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.070835 4191 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.070621 3814 consensus_queue.cc:1046] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:43.071602 3814 consensus_queue.cc:1225] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 03:43:43.072409 3905 raft_consensus.cc:3032] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:43.072810 3905 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:43.074056 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2301) took 1391 ms. Trace:
I20260806 03:43:43.073836 3905 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.074436 674 raft_consensus.cc:2396] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260806 03:43:43.074501 659 rpcz_store.cc:276] 0806 03:43:41.627007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.017139 (+1390132us) service_pool.cc:227] Handling call
0806 03:43:43.017508 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.017775 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:43.017804 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:43.017846 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:43.017881 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.017934 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.018208 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.075697 773 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:43.076427 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1298) took 2489 ms. Trace:
I20260806 03:43:43.076541 659 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 03:43:43.076781 773 rpcz_store.cc:276] 0806 03:43:40.586616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.074931 (+2488315us) service_pool.cc:227] Handling call
0806 03:43:43.075422 (+ 491us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:43.076152 (+ 730us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.076373 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.005686 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 762) took 1126 ms. Trace:
I20260806 03:43:43.077716 664 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 03:43:43.078086 664 pending_rounds.cc:76] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Aborting all ops after (but not including) 6
I20260806 03:43:43.004611 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:43.076876 674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1715) took 2600 ms (client timeout 3247 ms). Trace:
I20260806 03:43:43.061745 3883 consensus_queue.cc:260] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.079216 1668 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:43.046933 4000 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:43:43.005748 4048 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.078637 616 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.078362 664 pending_rounds.cc:97] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 03:43:43.048525 4130 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:43.082947 776 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:43.064373 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.076967 659 pending_rounds.cc:76] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Aborting all ops after (but not including) 5
I20260806 03:43:43.092579 659 pending_rounds.cc:97] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260806 03:43:43.080281 674 rpcz_store.cc:269] 0806 03:43:40.476178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.314259 (+838081us) service_pool.cc:227] Handling call
0806 03:43:43.076601 (+1762342us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540016dad4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.076814 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3961971072}
I20260806 03:43:43.033823 4164 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.094266 659 raft_consensus.cc:2866] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:43.019292 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:43.093984 664 raft_consensus.cc:2866] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:43.104097 499 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:43.087072 4000 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:43.104516 766 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:43.078008 1664 rpcz_store.cc:276] 0806 03:43:41.840861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.611934 (+771073us) service_pool.cc:227] Handling call
0806 03:43:42.612647 (+ 713us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:42.613272 (+ 625us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:43:42.770349 (+157077us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.966578 (+196229us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:42.966626 (+ 48us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:42.966671 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.966724 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:42.966820 (+ 96us) inbound_call.cc:173] Queueing success response
0806 03:43:43.055132 (+ 88312us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
Metrics: {"mutex_wait_us":64639,"prepare.queue_time_us":221816,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":52,"raft.queue_time_us":67391,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":3668,"spinlock_wait_cycles":26558720,"thread_start_us":199677,"threads_started":2,"wal-append.queue_time_us":137058}
I20260806 03:43:43.108297 661 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:43.109061 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 343) took 1325 ms. Trace:
I20260806 03:43:43.054234 3902 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.113273 563 rpcz_store.cc:276] 0806 03:43:41.779154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.018669 (+1239515us) service_pool.cc:227] Handling call
0806 03:43:43.104266 (+ 85597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.122721 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.125329 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1296) took 2591 ms. Trace:
I20260806 03:43:43.125771 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2293) took 1501 ms. Trace:
I20260806 03:43:43.125882 775 rpcz_store.cc:276] 0806 03:43:40.534040 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.729735 (+2195695us) service_pool.cc:227] Handling call
0806 03:43:42.808960 (+ 79225us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:42.809503 (+ 543us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.809532 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:42.845363 (+ 35831us) log.cc:842] Serialized 26 byte log entry
0806 03:43:42.982832 (+137469us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:42.982859 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.982945 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:43.125002 (+142057us) raft_consensus.cc:1671] finished
0806 03:43:43.125030 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.125212 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":657,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":164,"thread_start_us":119953,"threads_started":1,"wal-append.queue_time_us":260155}
I20260806 03:43:43.126433 668 rpcz_store.cc:276] 0806 03:43:41.624348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.588780 (+964432us) service_pool.cc:227] Handling call
0806 03:43:42.589462 (+ 682us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:42.642035 (+ 52573us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.642064 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:42.642654 (+ 590us) log.cc:842] Serialized 26 byte log entry
0806 03:43:42.840647 (+197993us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:42.893311 (+ 52664us) log.cc:842] Serialized 14 byte log entry
0806 03:43:42.920117 (+ 26806us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:42.920200 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:43.026048 (+105848us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b180038b120. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:43.125303 (+ 99255us) raft_consensus.cc:1671] finished
0806 03:43:43.125466 (+ 163us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.125652 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66539,"prepare.queue_time_us":53899,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":20373,"raft.queue_time_us":110048,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":16734,"spinlock_wait_cycles":4325632,"thread_start_us":85278,"threads_started":1,"wal-append.queue_time_us":193123}
I20260806 03:43:43.129859 1624 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:43.131055 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 119405us
W20260806 03:43:43.136628 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 47573.3us
W20260806 03:43:43.145653 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 46974.2us
I20260806 03:43:43.150020 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.121124 4255 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.143838 3883 raft_consensus.cc:3037] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:43.160876 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 82392.1us
W20260806 03:43:43.162393 1622 leader_election.cc:334] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.913s (SENT)
I20260806 03:43:43.166226 1622 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.169449 4141 raft_consensus.cc:2781] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:43.172256 4139 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.173082 4139 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.176427 661 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:43.178066 4141 raft_consensus.cc:686] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:43.181659 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:43:43.163707 3917 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.192159 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:43.194422 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 44524.5us
I20260806 03:43:43.194759 3613 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.198642 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.198952 4152 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260806 03:43:43.197074 4192 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.200407 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 758) took 1410 ms. Trace:
W20260806 03:43:43.201506 667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2257) took 3662 ms (client timeout 2446 ms). Trace:
I20260806 03:43:43.201921 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2274) took 3512 ms. Trace:
I20260806 03:43:43.202592 4000 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.205176 768 raft_consensus.cc:1226] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:43.179057 3980 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.214006 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1304) took 1873 ms. Trace:
I20260806 03:43:43.168826 4255 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.179958 4193 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:43:43.211310 667 rpcz_store.cc:269] 0806 03:43:39.538711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:39.539842 (+ 1131us) service_pool.cc:227] Handling call
0806 03:43:43.201293 (+3661451us) spinlock_profiling.cc:243] Waited 3.33 s on lock 0x7b540015a4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.201434 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7661163904}
I20260806 03:43:43.214363 768 rpcz_store.cc:276] 0806 03:43:41.340351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.204546 (+1864195us) service_pool.cc:227] Handling call
0806 03:43:43.204957 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.213735 (+ 8778us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.213938 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.210752 663 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:43.215296 767 raft_consensus.cc:1261] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:43.216636 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1308) took 1436 ms. Trace:
I20260806 03:43:43.216758 769 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:43.217006 767 rpcz_store.cc:276] 0806 03:43:41.779927 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.157220 (+1377293us) service_pool.cc:227] Handling call
0806 03:43:43.193024 (+ 35804us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.216372 (+ 23348us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.216573 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:43.217464 4141 consensus_queue.cc:237] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.211073 1660 rpcz_store.cc:276] 0806 03:43:41.789788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.414499 (+624711us) service_pool.cc:227] Handling call
0806 03:43:43.200079 (+785580us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400515754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.200311 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":532831104}
I20260806 03:43:43.221408 4000 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:43.179984 4114 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
I20260806 03:43:43.228264 4120 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.230544 4010 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:43.230813 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.237583 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:43.231276 4223 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 03:43:43.231782 4164 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.232234 770 raft_consensus.cc:3032] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:43.244920 770 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:43.234808 3905 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.235635 4057 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.238814 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.242153 1670 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:43.247695 775 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:43.248672 770 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.249719 770 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.248700 3901 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.254920 3902 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.246621 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.211908 671 rpcz_store.cc:276] 0806 03:43:39.689548 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:40.369164 (+679616us) service_pool.cc:227] Handling call
0806 03:43:40.369502 (+ 338us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:40.369833 (+ 331us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:40.369863 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:40.375596 (+ 5733us) log.cc:842] Serialized 52 byte log entry
0806 03:43:40.582257 (+206661us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:41.767960 (+1185703us) log.cc:842] Serialized 14 byte log entry
0806 03:43:41.798752 (+ 30792us) log.cc:842] Serialized 14 byte log entry
0806 03:43:43.044426 (+1245674us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b18001f5580. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7 00007f321633a058
0806 03:43:43.178058 (+133632us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.191345 (+ 13287us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:43.197392 (+ 6047us) raft_consensus.cc:1671] finished
0806 03:43:43.197419 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.201722 (+ 4303us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1147931,"prepare.queue_time_us":1083063,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":71,"raft.queue_time_us":74540,"raft.run_cpu_time_us":563,"raft.run_wall_time_us":64343,"spinlock_wait_cycles":14740992,"thread_start_us":404184,"threads_started":3,"wal-append.queue_time_us":278722,"wal-append.run_cpu_time_us":6164,"wal-append.run_wall_time_us":2165089}
W20260806 03:43:43.250121 4167 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 2.349s user 0.000s sys 0.003s
I20260806 03:43:43.255702 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.263687 770 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:43:43.264014 770 pending_rounds.cc:76] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 2
I20260806 03:43:43.264206 770 pending_rounds.cc:97] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:43:43.264407 770 raft_consensus.cc:2866] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:43.267400 1660 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:43.268743 4139 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.269949 768 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 3.
I20260806 03:43:43.270234 727 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.270625 4139 consensus_queue.cc:1046] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:43.271191 3980 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:43:43.271025 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:43.272591 3450 raft_consensus.cc:3032] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:43.273056 3450 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:43.273871 1668 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:43.274135 3450 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.279456 1624 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:43.274178 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.280738 769 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:43.281021 1624 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:43.281599 4251 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.679s user 0.001s sys 0.001s
I20260806 03:43:43.281811 4114 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:43.296998 4114 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.297371 4114 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.282851 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:43.298913 1665 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:43.282090 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:43.284060 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 106521us
I20260806 03:43:43.303701 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.304574 661 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:43.305294 3902 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:43.310436 4128 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:43.313338 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.286630 1660 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:43:43.315244 661 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:43.316004 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.291980 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:43.261042 4193 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:43.294704 4022 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:43.295483 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2299) took 1669 ms. Trace:
I20260806 03:43:43.310771 3980 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:43.334074 671 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:43.316938 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:43.275309 3450 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.319164 1624 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:43.289373 1622 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:43.328290 4139 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:43.331007 665 rpcz_store.cc:276] 0806 03:43:41.625983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.951262 (+1325279us) service_pool.cc:227] Handling call
0806 03:43:42.951723 (+ 461us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:42.977975 (+ 26252us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:42.978021 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:43.003540 (+ 25519us) log.cc:842] Serialized 52 byte log entry
0806 03:43:43.175766 (+172226us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:43:43.175804 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.175978 (+ 174us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:43.295230 (+119252us) raft_consensus.cc:1671] finished
0806 03:43:43.295259 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:43.295424 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13994,"raft.queue_time_us":26140,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":15899,"thread_start_us":140419,"threads_started":1,"wal-append.queue_time_us":177956}
I20260806 03:43:43.341693 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.348942 659 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:43.329573 4101 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:43.359606 667 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:43.352864 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.361305 4141 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:43.361871 4141 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:43.362387 4141 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.365720 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 319) took 2367 ms. Trace:
W20260806 03:43:43.366868 614 leader_election.cc:334] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.449s (SENT)
I20260806 03:43:43.367569 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:43.348176 3922 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:43.374156 3922 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.374634 3922 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.375319 4193 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.358742 3883 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260806 03:43:43.383185 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 121570us
I20260806 03:43:43.384784 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:43.343695 3651 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:43.386047 3651 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.386525 3651 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:43.388821 3907 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:43.388001 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 45308.9us
I20260806 03:43:43.366443 664 rpcz_store.cc:276] 0806 03:43:40.998486 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:41.450776 (+452290us) service_pool.cc:227] Handling call
0806 03:43:43.077083 (+1626307us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:43.120194 (+ 43111us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:43.365430 (+245236us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540016dad0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:43.365627 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3739316992}
W20260806 03:43:43.399523 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 47049.4us
I20260806 03:43:43.407992 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.356751 4101 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.351145 4152 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.339622 3902 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260806 03:43:43.345240 3901 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 03:43:43.422282 770 raft_consensus.cc:2362] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3. Re-sending same reply.
I20260806 03:43:43.423900 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.425550 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.414608 4101 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.412170 3917 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:43.427764 616 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.428817 776 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:43.429179 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 97877.4us
I20260806 03:43:43.420840 3980 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:43.417135 4010 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
W20260806 03:43:43.436069 4276 log.cc:924] Time spent T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20260806 03:43:43.438437 4098 consensus_queue.cc:1046] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:43.439368 4098 raft_consensus.cc:3032] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:43.439715 4098 raft_consensus.cc:729] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:43.440575 4098 consensus_queue.cc:260] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:43.446538 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 79863.3us
I20260806 03:43:43.425064 4255 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.435478 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:43.425989 4139 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:43.433184 4278 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.454798 563 raft_consensus.cc:1261] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:43.455375 4289 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:43.462420 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.465763 1670 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260806 03:43:43.475252 4280 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260806 03:43:43.478034 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:43:43.478976 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:43:43.496079 3901 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:43.498771 4119 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:43.499413 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.500145 769 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.500217 3917 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:43.501112 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.501856 767 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:43.458968 4255 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:43.502396 4255 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:43.502749 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.454818 4278 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.503441 773 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.503875 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.504310 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.504526 771 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.505095 4278 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.505599 4278 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.505523 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.419009 4152 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.514705 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.514945 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.515491 767 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:43.515735 772 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:43.515735 563 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:43.502779 4255 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.527590 767 raft_consensus.cc:1226] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:43.511530 4098 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.526263 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.544337 769 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.529235 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.506163 4278 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.551698 1671 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:43.457541 4114 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.548715 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.552053 544 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:43.555370 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 105814us
I20260806 03:43:43.553783 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.560039 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:43.562867 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.563529 664 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20260806 03:43:43.563808 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:43.564728 4057 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.564924 4061 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.489121 3922 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.569303 563 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.521584 4024 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 03:43:43.565516 4061 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.576109 1664 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:43.580423 4090 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
W20260806 03:43:43.586722 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 195806us
I20260806 03:43:43.610285 669 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:43.573324 4057 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.610600 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.618485 1622 leader_election.cc:398] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:43.629748 1622 leader_election.cc:401] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:43.629784 664 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:43.630643 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 45488.1us
I20260806 03:43:43.630797 4278 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.631419 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:43.620199 616 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.634063 616 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.622051 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.623557 498 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.566183 3905 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 03:43:43.624838 4000 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:43.625348 3883 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.628453 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:43.626171 4283 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260806 03:43:43.626988 547 raft_consensus.cc:1226] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:43.627326 4114 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.650525 562 raft_consensus.cc:1261] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:43.641827 4278 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:43.641880 3902 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:43.642930 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.631290 4139 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.618323 4024 consensus_queue.cc:1225] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:43:43.640959 4141 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.664423 616 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.672689 1658 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:43.672969 616 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.672806 3883 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.674127 4090 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:43.674635 4090 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.674958 4090 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.675320 3883 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:43.676576 3917 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:43.677624 616 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.672751 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.667881 769 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.654994 4278 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.665290 3902 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.666913 4010 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.718350 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.678574 3917 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:43.582264 3980 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.713496 616 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.716626 3902 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.713339 4278 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.696208 3814 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:43.727517 775 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:43:43.731768 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:43.689263 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:43.710611 4291 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:43.742636 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:43.665624 4000 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.728216 4268 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.755769 564 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.755148 616 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.727360 4057 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.744558 4010 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:43.710177 3651 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:43.741884 3917 raft_consensus.cc:686] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:43.775566 562 raft_consensus.cc:1226] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:43:43.749046 4291 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:43.755889 4130 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:43.783499 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.758930 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:43.788129 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 159769us
I20260806 03:43:43.784353 547 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:43.771037 4119 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:43.756603 1663 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.800535 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:43.777155 3922 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.804437 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.807297 775 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:43:43.806329 776 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 2. (term mismatch)
I20260806 03:43:43.805909 767 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
W20260806 03:43:43.806844 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 174915us
I20260806 03:43:43.809268 727 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:43.780827 4291 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:43.809973 616 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.807933 776 pending_rounds.cc:76] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 1
I20260806 03:43:43.817065 776 pending_rounds.cc:97] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 03:43:43.817410 776 raft_consensus.cc:2866] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:43.817767 776 pending_rounds.cc:97] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 03:43:43.818027 776 raft_consensus.cc:2866] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:43.794560 4061 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.797753 4119 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.830076 563 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:43.777336 4164 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 03:43:43.798872 4152 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:43.871445 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 73673.4us
I20260806 03:43:43.838559 547 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:43:43.847258 4271 log.cc:924] Time spent T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.611s user 0.007s sys 0.000s
W20260806 03:43:43.847770 4292 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
W20260806 03:43:43.850783 4297 log.cc:924] Time spent T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260806 03:43:43.854110 766 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:43:43.854559 4024 consensus_queue.cc:1046] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:43.856158 775 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.862673 3917 consensus_queue.cc:237] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.864499 3651 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.833591 3922 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:43.889328 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.892161 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:43.879236 4141 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.833947 3450 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:43.835626 4057 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.905862 4130 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260806 03:43:43.915005 4057 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.917255 4057 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.903082 3922 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:43.919057 3922 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:43.919337 4255 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:43.921562 4255 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:43.837858 4119 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:43.903510 563 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.904716 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:43.922917 776 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:43.956010 498 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:43.959677 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 67638.9us
I20260806 03:43:43.961730 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.922252 4288 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:43.975142 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:43.957368 4255 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:43.983475 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:43.981580 547 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:43.947013 4307 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:43.987002 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:43.963575 4000 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.005136 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.006541 547 raft_consensus.cc:2396] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:43:44.020973 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 178733us
I20260806 03:43:43.987528 4193 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:44.005393 4255 raft_consensus.cc:686] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:44.035125 1668 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:43:44.019014 615 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:44.047511 4200 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:43:44.049168 4101 consensus_meta.cc:287] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.606s user 0.012s sys 0.000s
I20260806 03:43:44.049647 4101 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.050546 4010 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:44.051007 4010 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:44.051429 4010 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:44.051939 4101 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.051661 4090 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.053836 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:44.055076 4300 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.149s user 0.000s sys 0.001s
I20260806 03:43:44.056756 1665 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:44.065389 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.065553 4255 consensus_queue.cc:237] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.066895 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.068964 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:44.069900 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.070837 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:44.080600 667 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:44.071142 4304 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260806 03:43:44.072074 661 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260806 03:43:44.021088 4288 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.072762 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:44.082823 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 126928us
I20260806 03:43:44.086833 564 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:44.091066 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.099917 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.100731 558 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:44.102656 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.078665 3651 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:43:44.078480 4200 consensus_queue.cc:1225] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:43:44.114295 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:44.086925 4090 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.129819 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.129154 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.137912 770 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:44.142306 546 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:44.131052 498 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
W20260806 03:43:44.127661 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 159767us
I20260806 03:43:44.153090 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.154673 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.155321 767 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:44.156281 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:44.158694 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48790us
I20260806 03:43:44.149713 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:44.167034 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:44.114635 4130 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:44.141621 3980 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:44.178226 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:44.129643 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.114795 4024 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260806 03:43:44.194021 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:44.195580 770 raft_consensus.cc:2396] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:44.197779 667 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:44.165194 3357 raft_consensus.cc:3032] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:43:44.200591 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 57356.7us
I20260806 03:43:44.186808 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.203655 776 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260806 03:43:44.209954 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 79006.7us
I20260806 03:43:44.150678 4152 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 03:43:44.150521 3883 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260806 03:43:44.218077 558 raft_consensus.cc:2396] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:43:44.227653 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 63213.7us
I20260806 03:43:44.228616 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:44.149766 4098 consensus_queue.cc:1035] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260806 03:43:44.189491 4000 raft_consensus.cc:2741] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 03:43:44.209635 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:44.155901 3907 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260806 03:43:44.187927 4139 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:44.165441 4090 consensus_queue.cc:1046] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260806 03:43:44.162900 4321 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260806 03:43:44.205071 3357 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:44.199641 3917 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.055s
I20260806 03:43:44.192862 4130 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:44.277999 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 68450.3us
W20260806 03:43:44.267695 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 50366.7us
W20260806 03:43:44.249614 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 110752us
I20260806 03:43:44.289248 4344 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.299759 4344 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.301990 4344 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.217767 4318 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:43:44.285064 4278 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.003s
I20260806 03:43:44.279524 4307 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:44.322450 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 91935.2us
I20260806 03:43:44.251499 4000 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:44.297134 4323 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20260806 03:43:44.336984 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:44.232311 4320 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.102s
I20260806 03:43:44.267534 4347 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260806 03:43:44.370545 3357 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.371912 3357 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:44.348812 3902 consensus_meta.cc:287] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.584s user 0.004s sys 0.008s
I20260806 03:43:44.372838 3902 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:44.268963 3905 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260806 03:43:44.374873 3902 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761)
I20260806 03:43:44.376161 3902 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.376582 3902 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.298921 401 catalog_manager.cc:5495] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 4 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:44.380787 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 131476us
W20260806 03:43:44.341928 4251 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 03:43:44.335214 3814 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:44.351514 4268 consensus_meta.cc:287] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.592s user 0.001s sys 0.011s
I20260806 03:43:44.386706 3651 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:44.387257 1622 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:44.367776 3907 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.350355 4024 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.389698 3814 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.406526 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:44.406898 4319 log.cc:924] Time spent T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260806 03:43:44.408231 3814 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:44.408726 4322 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260806 03:43:44.298635 4314 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.097s
I20260806 03:43:44.390462 4288 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.398447 4268 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:44.326782 4278 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.415548 3902 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.416518 4278 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.416895 3922 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.421355 1670 raft_consensus.cc:1226] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
W20260806 03:43:44.453647 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 66549.4us
I20260806 03:43:44.454749 1622 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:44.382753 4347 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:44.390587 4318 raft_consensus.cc:686] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:44.395283 4193 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:44.458137 546 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:44.459133 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:44.546989 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.459632 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.555603 776 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:44.531174 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.550845 562 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260806 03:43:44.459690 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 59609.3us
W20260806 03:43:44.557121 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 190501us
W20260806 03:43:44.556548 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 101895us
I20260806 03:43:44.560675 547 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:44.559911 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:44.560038 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:44.583038 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 440197us
W20260806 03:43:44.553457 4291 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.690s user 0.012s sys 0.001s
I20260806 03:43:44.457376 4278 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:44.536613 3906 raft_consensus.cc:3032] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:44.460062 4321 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:44.519394 4141 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:44.527864 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.421653 4090 consensus_queue.cc:1046] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.161s
W20260806 03:43:44.603349 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 44842.6us
I20260806 03:43:44.606769 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.613322 769 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:44.537519 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:44.588593 4010 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.537s user 0.004s sys 0.009s
I20260806 03:43:44.588112 4268 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:44.605733 614 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:44.609007 546 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:44.617206 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.417960 3907 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.604290 3906 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:44.578752 4120 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 03:43:44.606974 4278 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.631181 3906 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.632678 4278 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.633538 4318 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.636646 614 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:44.638396 544 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260806 03:43:44.638767 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:44.639303 3906 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:44.639478 561 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:44.456979 4024 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.642171 661 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:44.651743 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:44.643567 562 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:43:44.654887 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.655193 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:44.656709 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61257us
I20260806 03:43:44.638795 544 pending_rounds.cc:76] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 4
I20260806 03:43:44.657595 544 pending_rounds.cc:97] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 03:43:44.657863 544 raft_consensus.cc:2866] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:44.624308 4130 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.623093 4193 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.659222 548 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:43:44.586604 4347 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.661895 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.664857 659 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 03:43:44.661731 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.662156 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.656482 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.663852 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.619787 1660 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:44.605808 4291 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.666837 669 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:44.681677 769 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:44.625923 4164 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:44.626694 1658 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:44.669596 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:44.685051 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:44.685791 558 raft_consensus.cc:2362] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9. Re-sending same reply.
I20260806 03:43:44.669972 663 raft_consensus.cc:2379] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:44.654668 3922 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.670941 4120 raft_consensus.cc:2741] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260806 03:43:44.672736 4119 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.717s user 0.012s sys 0.000s
I20260806 03:43:44.702970 4119 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.705088 728 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:44.705464 4119 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.674229 664 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:44.706588 772 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:44.706856 667 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:44.637334 3907 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.674525 4193 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.674546 4192 raft_consensus.cc:3032] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:44.709373 4192 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:44.710412 4192 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.711474 4192 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:44.715626 3922 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:44.716161 4119 consensus_queue.cc:1046] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:44.689255 4164 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:44.718936 4363 raft_consensus.cc:3032] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:44.719331 4363 raft_consensus.cc:729] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:44.719754 4024 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.720090 4164 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.720194 4363 consensus_queue.cc:260] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:44.677985 4333 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 03:43:44.679013 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:44.680776 776 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:44.685070 3902 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:44.696614 4139 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:44.626880 4010 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:44.721215 402 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35374 (request call id 275) took 11585 ms (client timeout 15000 ms). Trace:
I20260806 03:43:44.723459 561 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:44.721724 401 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 3 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:44.753299 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 114569us
I20260806 03:43:44.754408 773 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:44.753334 767 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.580s user 0.009s sys 0.001s
I20260806 03:43:44.755439 767 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:44.760684 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:44.763978 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 184869us
I20260806 03:43:44.779047 665 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260806 03:43:44.780014 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:44.791000 401 catalog_manager.cc:5495] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 3, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 3 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:44.760869 4326 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:44.799422 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.800184 4326 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:44.812006 4326 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:44.812362 1622 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:44.813467 1658 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:44.814494 4164 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:44.814568 4268 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:44.814972 4164 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:44.815184 4268 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:44.815402 4164 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:44.815495 4010 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.808676 4101 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.816216 4268 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.816810 4101 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.818845 4101 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:44.808597 401 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 4 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:44.820742 4291 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:44.822007 4193 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.822196 4347 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:44.820847 401 catalog_manager.cc:5495] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 3, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:44.823150 4090 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:44.823697 4090 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.824218 1670 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 9: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:44.823848 401 catalog_manager.cc:5495] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:44.763406 3902 consensus_queue.cc:1035] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 03:43:44.825605 4090 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:44.793268 4370 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:44.827486 401 catalog_manager.cc:5495] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 4, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 4 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:44.810708 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:44.834120 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.863408 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.918649 401 catalog_manager.cc:5495] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:44.890389 542 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
W20260806 03:43:44.945634 4348 log.cc:924] Time spent T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.418s user 0.005s sys 0.008s
W20260806 03:43:44.959496 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 307863us
I20260806 03:43:44.963428 501 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:44.752278 402 rpcz_store.cc:269] 0806 03:43:33.135784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:33.138572 (+ 2788us) service_pool.cc:227] Handling call
0806 03:43:44.720997 (+11582425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:34.665665 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:34.939471 (+273806us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:34.960225 (+ 20754us) write_op.cc:435] Acquired schema lock
0806 03:43:34.989185 (+ 28960us) tablet.cc:660] Decoding operations
0806 03:43:35.495659 (+506474us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:35.514643 (+ 18984us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:35.527983 (+ 13340us) tablet.cc:683] Acquiring locks for 7 operations
0806 03:43:35.674601 (+146618us) tablet.cc:699] Row locks acquired
0806 03:43:35.674641 (+ 40us) write_op.cc:260] PREPARE: finished
0806 03:43:35.826617 (+151976us) write_op.cc:270] Start()
0806 03:43:35.913085 (+ 86468us) write_op.cc:276] Timestamp: P: 1785987815730589 usec, L: 0
0806 03:43:35.938954 (+ 25869us) op_driver.cc:331] REPLICATION: starting
0806 03:43:36.222583 (+283629us) log.cc:842] Serialized 2856 byte log entry
0806 03:43:36.740463 (+517880us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f32133d95fe 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb
0806 03:43:38.258491 (+1518028us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:39.781302 (+1522811us) op_driver.cc:444] REPLICATION: finished
0806 03:43:40.531871 (+750569us) write_op.cc:301] APPLY: starting
0806 03:43:40.544304 (+ 12433us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:40.584986 (+ 40682us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:40.585014 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:40.709075 (+124061us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:40.709455 (+ 380us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0806 03:43:40.709512 (+ 57us) write_op.cc:312] APPLY: finished
0806 03:43:40.709890 (+ 378us) log.cc:842] Serialized 59 byte log entry
0806 03:43:40.710052 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:40.710169 (+ 117us) write_op.cc:454] Released schema lock
0806 03:43:40.710226 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:40.710609 (+ 383us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:40.710640 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11170,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":558462,"apply.run_cpu_time_us":4774,"apply.run_wall_time_us":278964,"num_ops":7,"prepare.queue_time_us":547653,"prepare.run_cpu_time_us":39182,"prepare.run_wall_time_us":2329584,"raft.queue_time_us":955000,"raft.run_cpu_time_us":74823,"raft.run_wall_time_us":665174,"replication_time_us":3775628,"spinlock_wait_cycles":40851456,"thread_start_us":2437384,"threads_started":4,"wal-append.queue_time_us":839985,"wal-append.run_cpu_time_us":67953,"wal-append.run_wall_time_us":4846988}]]}
I20260806 03:43:44.972695 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:44.973711 1665 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260806 03:43:44.962153 1622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.715s (SENT) [suppressed 58 similar messages]
I20260806 03:43:44.976830 401 catalog_manager.cc:5495] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:43:44.983157 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:44.972127 3883 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:44.990477 563 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:44.995850 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 185286us
I20260806 03:43:44.996526 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1331) took 1388 ms. Trace:
I20260806 03:43:44.997382 4326 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:44.998009 614 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:44.981498 3906 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:44.992841 4344 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.001832 4344 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.003325 4363 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.003978 4344 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:45.005532 614 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.006556 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.007474 4090 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:45.008097 4090 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:45.008739 4379 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.088s user 0.002s sys 0.000s
I20260806 03:43:44.981151 3917 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:45.009369 3917 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.011498 3917 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:45.002239 4375 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:45.012284 4375 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:44.983875 4024 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:45.013218 4375 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.013357 4024 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.013931 4386 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:45.014549 4386 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.015565 4024 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:45.016618 4386 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:45.017386 4386 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.017838 4386 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.019732 4386 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:45.020954 3883 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.023447 4307 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:45.023937 4307 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:45.024313 4307 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:45.024552 4164 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.026029 4288 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:45.026886 4288 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.028564 3883 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:45.013744 1658 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:43:45.029031 4288 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:45.030347 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 325431us
I20260806 03:43:45.030634 4298 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:45.031122 4298 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:45.031471 4298 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:45.033432 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:45.033756 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:45.034030 775 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.034611 776 raft_consensus.cc:1261] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:45.034627 4331 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.035842 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.036134 4331 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.037221 1658 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
W20260806 03:43:45.023507 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 699452us
I20260806 03:43:45.038787 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.039470 770 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:45.039920 727 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.040755 4291 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:43:45.043174 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.044297 775 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:44.996929 771 rpcz_store.cc:276] 0806 03:43:43.607624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:43.607745 (+ 121us) service_pool.cc:227] Handling call
0806 03:43:43.626863 (+ 19118us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:43.632814 (+ 5951us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:43.632852 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:43.651765 (+ 18913us) log.cc:842] Serialized 26 byte log entry
0806 03:43:44.217162 (+565397us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b180038b6c0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa6411 00007f32162f35cf
0806 03:43:44.321225 (+104063us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:44.321285 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:44.321492 (+ 207us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:44.944969 (+623477us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:44.996293 (+ 51324us) raft_consensus.cc:1671] finished
0806 03:43:44.996312 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:44.996431 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5552,"raft.queue_time_us":26792,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":8989,"spinlock_wait_cycles":320546944,"thread_start_us":103763,"threads_started":1,"wal-append.queue_time_us":784160}
I20260806 03:43:45.045946 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.049957 728 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:44.993295 401 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 3, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 3 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:45.051796 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 58471.2us
I20260806 03:43:45.055029 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:45.055316 498 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.058665 401 catalog_manager.cc:5495] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 3, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.059351 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:45.060128 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.050239 776 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:45.067238 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:45.049908 4061 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.012s sys 0.017s
I20260806 03:43:45.073386 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:45.056306 4291 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:45.089967 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.074158 1668 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:43:45.084582 1665 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:45.105243 1658 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:45.104900 616 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.082731 401 catalog_manager.cc:5495] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 3, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 3 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.056218 4331 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:45.071961 4061 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.967s user 0.012s sys 0.017s
I20260806 03:43:45.070199 4344 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:45.169750 616 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.058570 3906 consensus_queue.cc:1046] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.111s
I20260806 03:43:45.073551 4350 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.180s
I20260806 03:43:45.405702 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:43:45.293625 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 331469us
I20260806 03:43:45.235098 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.232924 771 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:45.249399 669 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:45.219750 401 catalog_manager.cc:5495] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
W20260806 03:43:45.515201 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 443526us
I20260806 03:43:45.291368 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.521406 498 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:45.553320 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 383648us
I20260806 03:43:45.559948 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.127853 4356 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:43:45.171671 4385 log.cc:924] Time spent T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20260806 03:43:45.469420 4387 log.cc:924] Time spent T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.341s user 0.000s sys 0.001s
I20260806 03:43:45.521616 770 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:45.172885 615 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.564416 401 catalog_manager.cc:5495] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.123804 4291 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:45.566883 616 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.130263 4278 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.479295 4164 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:45.578769 401 catalog_manager.cc:5495] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 4, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.420763 4326 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:45.571319 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 400434us
I20260806 03:43:45.165766 4139 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:45.334298 4337 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260806 03:43:45.174211 4098 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.401s
W20260806 03:43:45.243673 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:45.169706 4289 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:45.580801 401 catalog_manager.cc:5495] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 2 to 4, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.291708 4307 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.571010 1622 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
W20260806 03:43:45.578755 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 405930us
I20260806 03:43:45.569537 4376 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:45.169286 3980 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.407s
I20260806 03:43:45.593293 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.566206 4380 raft_consensus.cc:2781] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:45.311204 4193 consensus_queue.cc:1046] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.262s
I20260806 03:43:45.595340 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.573092 4291 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:45.583825 4390 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.299s user 0.001s sys 0.000s
I20260806 03:43:45.579704 4061 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:45.577491 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.519127 4331 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:45.623361 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.623438 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:45.675372 1664 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.587193 4375 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:45.584154 3906 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260806 03:43:45.622793 4139 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.582769 4278 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.574074 4057 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.395826 4368 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:45.656982 376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 437385us
I20260806 03:43:45.661170 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.634912 4326 consensus_queue.cc:1046] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260806 03:43:45.659718 401 catalog_manager.cc:5495] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 reported cstate change: term changed from 1 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } }
I20260806 03:43:45.666013 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.669466 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.668677 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.669183 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.582546 4356 raft_consensus.cc:686] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:45.674899 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.675521 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:45.677942 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 100566us
I20260806 03:43:45.659389 4289 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.678828 4375 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:45.669512 3980 consensus_queue.cc:1225] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 03:43:45.666582 4119 consensus_queue.cc:1046] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260806 03:43:45.679315 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:45.689267 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 96206.2us
I20260806 03:43:45.710068 564 raft_consensus.cc:2354] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:45.759917 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:45.678351 4411 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.711577 544 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:45.694483 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.712880 547 raft_consensus.cc:2379] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:45.713812 562 raft_consensus.cc:2354] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:45.677735 4380 raft_consensus.cc:686] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:45.693323 4412 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260806 03:43:45.685165 4061 raft_consensus.cc:686] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:45.805250 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.691401 3902 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:45.755511 4356 consensus_queue.cc:237] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.756532 4130 raft_consensus.cc:3032] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:45.757764 4307 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:45.753742 616 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.786625 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.789142 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.743916 3906 consensus_queue.cc:1225] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:43:45.752820 4375 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:45.684296 4331 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.838493 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.802927 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.692485 4120 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:45.842176 547 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:45.842428 4139 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:45.843218 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.843714 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.844476 562 raft_consensus.cc:2354] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:45.781252 4000 raft_consensus.cc:3032] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:45.808769 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.810864 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:45.792973 4400 log.cc:924] Time spent T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.224s user 0.001s sys 0.001s
W20260806 03:43:45.798303 4255 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.185s user 0.002s sys 0.025s
I20260806 03:43:45.787470 4200 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:45.818774 558 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
W20260806 03:43:45.829089 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 75438.5us
I20260806 03:43:45.773567 4098 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 03:43:45.801416 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.857617 616 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:45.858125 548 raft_consensus.cc:2362] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5. Re-sending same reply.
W20260806 03:43:45.858575 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 201900us
I20260806 03:43:45.844411 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.809340 3902 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:45.821030 4130 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:45.712970 4368 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:45.846748 4399 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260806 03:43:45.851073 544 raft_consensus.cc:2354] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:45.853448 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.865110 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.853806 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:45.836009 4119 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:45.866755 4119 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:45.867157 4119 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.854722 4388 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:45.861052 547 raft_consensus.cc:2354] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4.
I20260806 03:43:45.847560 1658 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:45.852807 4255 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.245s user 0.003s sys 0.025s
I20260806 03:43:45.840140 4363 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.879791 4363 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.839771 4307 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:45.882570 4307 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:45.883495 4413 log.cc:924] Time spent T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20260806 03:43:45.885525 4200 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:45.886026 4200 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:45.886581 4200 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:45.888096 4061 consensus_queue.cc:237] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.866313 4351 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:45.876549 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.898307 548 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 9: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:45.898963 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:45.856086 4398 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260806 03:43:45.901582 1664 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:45.878404 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.878659 4130 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.879145 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.848096 4000 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:45.864821 3902 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:45.885974 4404 log.cc:924] Time spent T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.209s user 0.000s sys 0.000s
I20260806 03:43:45.909301 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.910274 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.910789 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:45.911484 671 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:45.911434 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.912628 4289 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:45.912588 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.913447 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.913744 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.914763 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:45.915614 773 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:45.900444 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.877549 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:45.864367 727 connection.cc:232] RPC call timeout handler was delayed by -3.163589428178966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3247s for a response
I20260806 03:43:45.902247 544 raft_consensus.cc:3037] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.788408 4411 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.857378 4422 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260806 03:43:45.902761 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.903340 547 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:45.906242 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.873574 4326 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:45.921747 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.906939 4351 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:45.906742 4374 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:45.915380 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.924425 775 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:45.899971 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.891944 4363 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:45.922937 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.929471 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.916747 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.925516 548 raft_consensus.cc:2379] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:45.932173 563 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:45.932339 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.934877 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.934590 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.933092 548 raft_consensus.cc:2354] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:45.935842 546 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:45.934811 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.935756 547 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:45.937722 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.938788 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:45.939515 669 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:45.939821 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:45.940032 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.940629 770 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:43:45.939199 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:45.941816 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.944842 614 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:45.890585 4307 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.946511 4307 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:45.948679 4000 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:45.842240 4380 consensus_queue.cc:237] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:45.955299 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:45.940888 665 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:45.964843 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.965876 671 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:45.966830 562 raft_consensus.cc:1261] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:43:45.930832 4255 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:45.969564 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.972965 4119 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.974000 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.999001 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:45.935880 4130 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:45.998826 544 raft_consensus.cc:2454] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
W20260806 03:43:45.978591 4298 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.947s user 0.011s sys 0.003s
I20260806 03:43:45.980099 727 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:45.999475 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:45.938696 4363 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 03:43:46.025171 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 54903.2us
I20260806 03:43:45.964288 4351 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:46.027451 4408 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.393s user 0.002s sys 0.002s
I20260806 03:43:45.943133 4278 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:45.935751 4098 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:45.999089 546 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:46.019767 768 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:46.020813 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 796) took 1049 ms. Trace:
I20260806 03:43:45.999856 668 raft_consensus.cc:2396] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:46.035166 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:45.929539 4326 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:45.963706 4423 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:46.034665 1670 rpcz_store.cc:276] 0806 03:43:44.971122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.971307 (+ 185us) service_pool.cc:227] Handling call
0806 03:43:45.011508 (+ 40201us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:45.011707 (+ 199us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:45.011909 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:43:45.127672 (+115763us) log.cc:842] Serialized 14 byte log entry
0806 03:43:45.187298 (+ 59626us) log.cc:842] Serialized 14 byte log entry
0806 03:43:45.934253 (+746955us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b18003fbce0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f321336e4ca 00007f321ca4ff99 00007f3213232990 00007f32133f4864 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7 00007f321633a058
0806 03:43:46.011230 (+ 76977us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:46.011292 (+ 62us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:46.011358 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:46.020594 (+ 9236us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:46.020740 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":747858,"prepare.queue_time_us":446282,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":77,"raft.queue_time_us":773838,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":17160,"spinlock_wait_cycles":102292992,"thread_start_us":191643,"threads_started":2}
I20260806 03:43:46.055073 770 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:46.059242 546 raft_consensus.cc:1261] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:43:45.940131 4374 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.076273 4438 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:46.078111 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 121845us
I20260806 03:43:46.038236 4363 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.036841 4139 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:46.095527 616 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:46.080063 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 135251us
I20260806 03:43:46.101634 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:46.100021 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 46569.1us
I20260806 03:43:46.103253 616 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:46.042328 4291 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.106568 546 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:46.103905 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:46.113296 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:46.114197 614 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:46.044631 4289 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260806 03:43:46.117468 4363 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:46.023164 4298 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.100862 4438 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.109926 4368 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.187062 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:46.105862 4278 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.109181 4425 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:46.331507 769 raft_consensus.cc:1226] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:46.178602 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.120513 4363 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:46.332540 775 raft_consensus.cc:2396] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:46.304502 4446 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.111491 4331 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.336997 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:46.120582 4411 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:46.340006 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 225873us
I20260806 03:43:46.168071 4200 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.338032 4119 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:46.339148 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 225960us
I20260806 03:43:46.334676 4363 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:46.210388 4375 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.339044 4291 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.345745 4061 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:46.346791 4368 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:46.368243 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.368533 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:46.365626 4445 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:46.368544 4411 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.368862 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.369294 4374 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.370978 4200 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.371236 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:46.411278 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:46.411648 772 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.411221 665 raft_consensus.cc:2379] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:46.335155 3902 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:46.376494 4450 log.cc:924] Time spent T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.282s user 0.001s sys 0.000s
I20260806 03:43:46.379402 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.381537 667 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:43:46.359746 4417 raft_consensus.cc:3032] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:46.387158 669 raft_consensus.cc:2396] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:46.391551 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.387157 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:46.422758 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61076.5us
I20260806 03:43:46.423995 727 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.425398 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:46.406163 770 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:46.389868 661 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:46.426039 4289 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.406244 775 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:46.358050 4351 consensus_queue.cc:1035] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260806 03:43:46.367241 4061 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:46.427392 4061 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.429389 4289 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:46.429958 4289 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:46.430394 4289 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.432556 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.434478 4291 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.434717 4255 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:43:46.440420 727 leader_election.cc:398] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:46.442518 727 leader_election.cc:401] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:46.443236 4331 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:46.444646 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.420250 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.420934 4417 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:46.448441 3902 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.448315 4417 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.449033 4298 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.452255 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.452932 771 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:46.453387 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:46.419328 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 46519.8us
I20260806 03:43:46.454406 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.455718 544 raft_consensus.cc:2379] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:46.455974 498 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.457365 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.454780 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.454061 546 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:46.416412 4326 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.460150 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:46.458871 671 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:46.464506 498 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.467031 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:46.472759 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:46.474256 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 132444us
I20260806 03:43:46.475693 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:46.343966 4446 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.478022 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.478657 544 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:46.481972 4446 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.482620 4278 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:43:46.484018 4380 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:46.484443 4380 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:46.484810 4380 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.485608 775 raft_consensus.cc:1261] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260806 03:43:46.486228 4409 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.823s user 0.009s sys 0.000s
I20260806 03:43:46.486425 4164 raft_consensus.cc:3032] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:46.486891 4164 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:46.487468 4414 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:46.484812 4438 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.487893 4164 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.489234 4164 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:46.491598 3883 raft_consensus.cc:3032] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:46.492008 3883 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:46.492844 3883 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.494685 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 345) took 1390 ms. Trace:
I20260806 03:43:46.495213 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:46.496229 564 raft_consensus.cc:2379] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:46.497102 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2320) took 1856 ms. Trace:
I20260806 03:43:46.497326 564 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:46.497537 4461 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.497484 674 rpcz_store.cc:276] 0806 03:43:44.641094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.656796 (+ 15702us) service_pool.cc:227] Handling call
0806 03:43:44.667350 (+ 10554us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:44.720595 (+ 53245us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:44.720956 (+ 361us) log.cc:842] Serialized 14 byte log entry
0806 03:43:44.983049 (+262093us) log.cc:842] Serialized 14 byte log entry
0806 03:43:44.986261 (+ 3212us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:44.986323 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:44.986398 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:45.755976 (+769578us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b48001a01d8. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:43:46.450656 (+694680us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b18001f6a80. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133c4f96 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9
0806 03:43:46.496873 (+ 46217us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b540036c9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:46.496914 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:46.497043 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57644,"prepare.queue_time_us":1932,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":1980,"raft.queue_time_us":64991,"raft.run_cpu_time_us":3759,"raft.run_wall_time_us":773007,"spinlock_wait_cycles":132740096,"thread_start_us":201299,"threads_started":1,"wal-append.queue_time_us":332689}
I20260806 03:43:46.499383 4461 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:46.499750 4461 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:46.495049 659 rpcz_store.cc:276] 0806 03:43:45.104077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.104249 (+ 172us) service_pool.cc:227] Handling call
0806 03:43:45.104712 (+ 463us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:45.109706 (+ 4994us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:45.109748 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:45.164162 (+ 54414us) log.cc:842] Serialized 26 byte log entry
0806 03:43:45.639209 (+475047us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:45.663114 (+ 23905us) log.cc:842] Serialized 14 byte log entry
0806 03:43:45.672731 (+ 9617us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:45.672834 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:46.393572 (+720738us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18001822c0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7764 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:46.410803 (+ 17231us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b18001822c0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:46.489902 (+ 79099us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:46.494449 (+ 4547us) raft_consensus.cc:1671] finished
0806 03:43:46.494469 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:46.494619 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24549,"prepare.queue_time_us":25780,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":49,"raft.queue_time_us":20118,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":54715,"spinlock_wait_cycles":43471744,"thread_start_us":395093,"threads_started":1,"wal-append.queue_time_us":465012}
I20260806 03:43:46.500077 4461 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:46.459777 4428 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.498077 564 raft_consensus.cc:595] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:46.504031 564 raft_consensus.cc:603] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:46.516476 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.535733 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:46.536509 770 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.469182 4298 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:46.417328 4411 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.487893 4414 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:46.525575 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.494263 3883 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:46.559072 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 114537us
I20260806 03:43:46.560330 614 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:46.567054 727 leader_election.cc:398] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:46.567427 727 leader_election.cc:401] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:43:46.475886 4422 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:46.570905 562 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:43:46.576851 564 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:46.576787 772 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:46.581338 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:46.500010 3814 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:46.536903 4446 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.536600 4331 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:46.540846 4298 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:46.590131 544 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:46.567855 4445 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:46.535389 4119 consensus_queue.cc:1046] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260806 03:43:46.535966 4386 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 03:43:46.586856 4425 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:43:46.608968 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 49019.9us
W20260806 03:43:46.589570 4307 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.643s user 0.011s sys 0.001s
I20260806 03:43:46.615775 770 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:46.539495 4061 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.569774 4130 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260806 03:43:46.570006 4374 consensus_queue.cc:1035] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 03:43:46.598747 4298 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:46.628400 561 raft_consensus.cc:1261] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:46.652333 727 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.618112 4192 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.653600 4307 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:46.660881 4344 raft_consensus.cc:3032] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:46.662160 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 84264us
W20260806 03:43:46.627547 542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 353) took 2262 ms (client timeout 1715 ms). Trace:
I20260806 03:43:46.543036 4428 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.593364 4446 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.720623 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.721381 669 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:46.650549 4375 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.620282 4472 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:46.723379 674 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:46.658063 4255 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.602569 4331 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.736409 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.737313 671 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:46.725488 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1348) took 2293 ms. Trace:
I20260806 03:43:46.740511 616 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:46.726112 4057 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.032s user 0.015s sys 0.007s
I20260806 03:43:46.668493 4344 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:46.629863 4380 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.728569 4423 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.739401 4255 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:46.748310 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 131141us
I20260806 03:43:46.619901 4414 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:46.717338 542 rpcz_store.cc:269] 0806 03:43:44.364746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.366655 (+ 1909us) service_pool.cc:227] Handling call
0806 03:43:46.627286 (+2260631us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b5400086454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:46.627423 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541408000}
W20260806 03:43:46.749128 4474 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260806 03:43:46.725910 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:46.730793 4375 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:46.755213 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:46.727705 4471 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:46.758769 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.715902 4192 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.756681 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.751358 4307 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:46.764153 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:46.780076 564 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.738461 766 rpcz_store.cc:276] 0806 03:43:44.432561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.433752 (+ 1191us) service_pool.cc:227] Handling call
0806 03:43:44.501882 (+ 68130us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:45.034824 (+532942us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:45.034852 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:45.035261 (+ 409us) log.cc:842] Serialized 26 byte log entry
0806 03:43:45.171619 (+136358us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:45.559116 (+387497us) log.cc:842] Serialized 14 byte log entry
0806 03:43:46.500846 (+941730us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b18002ff140. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7 00007f321633a058
0806 03:43:46.725166 (+224320us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:46.725250 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:46.725293 (+ 43us) raft_consensus.cc:1671] finished
0806 03:43:46.725304 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:46.725430 (+ 126us) inbound_call.cc:173] Queueing success response
0806 03:43:46.728892 (+ 3462us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b18002ff140. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
Metrics: {"mutex_wait_us":509790,"prepare.queue_time_us":50974,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":1732,"raft.queue_time_us":532891,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":28207616,"thread_start_us":357894,"threads_started":2,"wal-append.queue_time_us":468021}
I20260806 03:43:46.766147 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.742581 4472 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:46.768539 4015 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:46.625891 4445 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.790441 4380 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.790578 4446 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.792106 4291 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:46.769074 562 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:46.769362 4475 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.134s user 0.000s sys 0.003s
I20260806 03:43:46.778306 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.778641 614 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:46.762274 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.771898 1660 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:46.786617 4455 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.298s user 0.000s sys 0.001s
I20260806 03:43:46.787158 4476 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:46.788728 4470 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 03:43:46.789335 4478 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:46.765977 4057 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:46.778856 4465 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.785501 4331 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:46.839291 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53584.6us
W20260806 03:43:46.841404 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 62912.6us
I20260806 03:43:46.840984 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.792809 4291 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:46.872258 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 407811us
I20260806 03:43:46.876189 768 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:46.877125 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:46.780570 4411 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:46.878746 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 118064us
I20260806 03:43:46.883247 546 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:46.883678 727 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.783844 4061 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:46.879439 547 raft_consensus.cc:1261] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:46.866647 3814 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:46.895911 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:46.906298 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2235) took 1845 ms. Trace:
I20260806 03:43:46.793895 4414 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:46.885900 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.778568 4375 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.891014 4469 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:46.891249 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.891433 727 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:46.892155 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.782034 4471 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.893464 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:46.876875 4476 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.909227 769 rpcz_store.cc:276] 0806 03:43:45.060524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.060654 (+ 130us) service_pool.cc:227] Handling call
0806 03:43:45.071487 (+ 10833us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:46.331985 (+1260498us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:46.905946 (+573961us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001801d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:46.906216 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2897360256}
I20260806 03:43:46.916782 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:46.794260 4344 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.927983 542 raft_consensus.cc:1261] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 03:43:46.837023 4015 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.939782 546 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:46.913492 3357 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:46.971525 3357 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.973476 4015 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.973747 3357 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:46.977555 4417 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:46.980928 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 89719us
I20260806 03:43:46.914234 4061 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:46.904346 3814 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.878002 4465 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:46.946691 558 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
W20260806 03:43:47.006605 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 131875us
I20260806 03:43:47.063442 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1822) took 1403 ms. Trace:
I20260806 03:43:46.951803 4291 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:46.987764 564 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:47.083060 563 rpcz_store.cc:276] 0806 03:43:45.660002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.660169 (+ 167us) service_pool.cc:227] Handling call
0806 03:43:47.063191 (+1403022us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b5400179654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:47.063364 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":500187008}
W20260806 03:43:47.026448 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 44765.5us
I20260806 03:43:46.956444 4130 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:47.090584 4098 raft_consensus.cc:3032] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:43:47.090823 1660 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:47.093607 4476 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.094702 4471 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.095021 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:47.095796 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:47.096946 775 raft_consensus.cc:1261] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 03:43:47.098748 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:47.099685 4428 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:47.101665 4164 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.612s user 0.009s sys 0.002s
W20260806 03:43:47.104228 4459 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.657s user 0.002s sys 0.000s
I20260806 03:43:46.965677 4255 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.094732 768 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:47.113844 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:46.970959 542 pending_rounds.cc:76] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 1
I20260806 03:43:47.111413 767 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:47.114524 773 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:47.113003 775 raft_consensus.cc:1226] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:47.028021 4344 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:47.031718 4015 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.043140 4192 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:47.132234 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 236430us
I20260806 03:43:47.135169 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:47.135478 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46849.3us
I20260806 03:43:47.135828 766 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:47.136651 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:47.109184 4098 raft_consensus.cc:729] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:47.137344 775 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:47.119282 542 pending_rounds.cc:97] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260806 03:43:47.125408 4461 consensus_meta.cc:287] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.621s user 0.013s sys 0.000s
W20260806 03:43:47.139089 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 129842us
I20260806 03:43:47.040885 4298 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:47.140901 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:47.138633 498 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:46.990964 4061 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.013121 4469 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260806 03:43:47.153151 4061 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.154246 674 raft_consensus.cc:2396] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:47.131537 4015 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:46.969404 4472 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.155122 4375 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.155781 4298 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:47.157104 772 raft_consensus.cc:1226] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 3. Current term is 4. Ops: []
W20260806 03:43:47.157210 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.5862e+06us
I20260806 03:43:47.161477 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.143205 542 raft_consensus.cc:2866] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:47.157104 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:47.160146 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.162209 775 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:43:47.162664 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:47.164099 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.171188 767 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 03:43:47.164408 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:47.171610 767 pending_rounds.cc:76] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 7
I20260806 03:43:47.171904 767 pending_rounds.cc:97] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260806 03:43:47.172148 767 raft_consensus.cc:2866] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:47.165319 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:47.165936 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:47.131852 1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 402) took 2287 ms (client timeout 2737 ms). Trace:
I20260806 03:43:47.170625 773 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:47.178901 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:47.193256 1658 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:47.166872 4326 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:47.175928 4363 consensus_meta.cc:287] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.810s user 0.012s sys 0.001s
I20260806 03:43:47.187793 4114 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:47.196870 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:47.197005 4498 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.179157 4446 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:47.199246 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:47.211100 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 48515.7us
I20260806 03:43:47.177572 3613 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.173195 4318 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:47.232878 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.382233 546 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:47.203305 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:47.237572 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 76167us
I20260806 03:43:47.383854 1665 raft_consensus.cc:2396] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:47.188040 4427 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:47.179258 4374 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:47.274111 4498 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.302670 3613 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.238289 4255 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:43:47.324752 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 89419.5us
I20260806 03:43:47.287019 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.324256 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:47.199131 4152 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:47.239497 3814 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.318601 3922 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:47.409740 4255 raft_consensus.cc:686] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:47.410068 3922 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:47.398305 4374 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:47.412643 4374 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:47.179538 1660 rpcz_store.cc:269] 0806 03:43:44.831371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.851072 (+ 19701us) service_pool.cc:227] Handling call
0806 03:43:46.670285 (+1819213us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:47.118879 (+448594us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10134784}
I20260806 03:43:47.329012 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.325482 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2333) took 1726 ms. Trace:
I20260806 03:43:47.168032 4483 consensus_queue.cc:1035] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.143s
W20260806 03:43:47.289723 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 133086us
I20260806 03:43:47.238078 4363 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:47.289959 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 76991.4us
W20260806 03:43:47.269266 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 133052us
I20260806 03:43:47.428933 727 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:47.289372 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 49552.9us
W20260806 03:43:47.300014 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 106720us
I20260806 03:43:47.187940 4291 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:47.235221 4464 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:47.235301 4114 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.440380 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:47.145255 4461 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:47.281446 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 102596us
I20260806 03:43:47.183539 4193 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:47.323104 4015 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.157433 4200 consensus_queue.cc:1046] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260806 03:43:47.321024 4298 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:47.420107 4465 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.384244 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.235651 4101 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:47.393160 499 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.240464 4446 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:47.240227 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:47.202399 4480 consensus_queue.cc:1046] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.191s
I20260806 03:43:47.409250 4505 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.410714 3922 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:47.326726 4354 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:47.414516 661 rpcz_store.cc:276] 0806 03:43:45.599505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.599660 (+ 155us) service_pool.cc:227] Handling call
0806 03:43:45.622346 (+ 22686us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:46.426467 (+804121us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:47.325271 (+898804us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54000663d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:47.325425 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758209792}
I20260806 03:43:47.414342 547 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:47.427631 1670 raft_consensus.cc:2379] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:47.193279 4490 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.206s
I20260806 03:43:47.433686 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:47.438884 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:47.395190 4268 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260806 03:43:47.449150 4278 consensus_queue.cc:1035] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.129s
I20260806 03:43:47.543138 766 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260806 03:43:47.600039 4501 log.cc:924] Time spent T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20260806 03:43:47.576138 4291 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.619831 4291 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.620898 4000 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:47.622876 4502 log.cc:924] Time spent T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
I20260806 03:43:47.623330 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:47.624816 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 240642us
I20260806 03:43:47.616930 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.626865 564 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:47.628170 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.445405 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2245) took 2273 ms. Trace:
I20260806 03:43:47.197480 4098 consensus_queue.cc:260] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:47.446267 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:47.639364 671 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 9.
I20260806 03:43:47.446805 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:47.447806 616 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:47.223277 563 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "d8469561a52c4bbc81dce786f1211ee6"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:47.427258 4152 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.461459 771 raft_consensus.cc:2396] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:43:47.398120 4427 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:47.649677 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 206483us
I20260806 03:43:47.650810 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:47.557351 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 128490us
I20260806 03:43:47.671802 563 raft_consensus.cc:595] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Received request to transfer leadership to TS d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.672374 563 raft_consensus.cc:603] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:47.672971 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:47.443696 3980 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:47.448997 4061 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:47.447299 4331 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:47.701568 664 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:47.694514 4015 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:47.711499 4015 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:47.711752 4493 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:47.426344 4326 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:47.577096 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:47.646867 4414 raft_consensus.cc:3032] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:47.751713 4518 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:47.722021 3980 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:47.752806 4518 consensus_queue.cc:1225] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer 0374a84d5e9f4e3c8d8287e2382379c8 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 03:43:47.752100 4268 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:47.753337 4508 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260806 03:43:47.754206 4289 consensus_meta.cc:287] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.323s user 0.014s sys 0.000s
I20260806 03:43:47.722494 4493 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:47.455438 4298 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:47.754930 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 81976.6us
I20260806 03:43:47.707161 4061 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:47.707161 4476 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.712160 4114 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.737478 4015 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:47.673404 4152 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.553050 1664 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:47.543068 4101 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.754669 4289 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:47.756042 727 leader_election.cc:398] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6.
I20260806 03:43:47.462846 4446 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:47.596501 4255 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.757910 4493 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:47.759563 4061 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:47.559422 4505 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.622113 4291 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.755491 3902 consensus_queue.cc:1046] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260806 03:43:47.456004 4503 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:47.815714 4498 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.836186 4503 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.839565 4291 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:47.840471 4291 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:47.753818 4268 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:47.841260 4503 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.749352 4414 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:47.804133 4318 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:47.841878 4503 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.842077 4318 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:47.842468 4503 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.770690 3613 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.834450 4505 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:47.770624 4476 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.844743 4503 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:47.846387 4476 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.846946 4476 raft_consensus.cc:3032] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:47.638792 776 rpcz_store.cc:276] 0806 03:43:45.172222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.215343 (+ 43121us) service_pool.cc:227] Handling call
0806 03:43:45.271978 (+ 56635us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:45.272245 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:43:45.638211 (+365966us) log.cc:842] Serialized 14 byte log entry
0806 03:43:45.788115 (+149904us) log.cc:842] Serialized 14 byte log entry
0806 03:43:47.363157 (+1575042us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b1800317da0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7 00007f321633a058
0806 03:43:47.445053 (+ 81896us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:47.445104 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:47.445156 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:47.445237 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:47.445348 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":218373,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":74,"raft.queue_time_us":110233,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":137,"spinlock_wait_cycles":24132480,"thread_start_us":556563,"threads_started":3,"wal-append.queue_time_us":192908,"wal-append.run_cpu_time_us":1706,"wal-append.run_wall_time_us":1048558}
I20260806 03:43:47.847597 4476 raft_consensus.cc:729] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:47.848963 4427 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:47.847180 4414 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:47.851902 4414 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:47.852296 4363 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:47.887323 4152 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.844203 4291 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:47.898777 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 466420us
W20260806 03:43:47.909343 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 171742us
I20260806 03:43:47.923806 770 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:47.813688 4461 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:47.899946 4152 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
W20260806 03:43:47.928942 4344 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.794s user 0.006s sys 0.007s
I20260806 03:43:47.929610 4510 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:48.035782 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 338769us
I20260806 03:43:48.004707 4363 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:47.999320 542 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:47.999722 4510 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:48.109010 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 483303us
I20260806 03:43:48.110215 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:48.111335 4152 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:43:48.111994 4152 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:48.113154 4152 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.098552 4101 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.115911 4482 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.118616 4482 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.122598 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 729792us
I20260806 03:43:48.077651 4318 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.177862 4510 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.121029 4464 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:48.178794 4464 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:48.180774 4439 log.cc:924] Time spent T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 1.769s user 0.011s sys 0.000s
I20260806 03:43:48.182940 4326 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:48.183544 4326 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.185616 4484 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 1.197s user 0.010s sys 0.005s
I20260806 03:43:48.175746 4298 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.081977 4363 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.155390 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 45304.8us
I20260806 03:43:48.230070 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36546 (request call id 492) took 1313 ms. Trace:
I20260806 03:43:48.230710 669 rpcz_store.cc:276] 0806 03:43:46.916289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:46.937553 (+ 21264us) service_pool.cc:227] Handling call
0806 03:43:48.227085 (+1289532us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163257c5 00007f321631dc1a 00007f321b2616d9 00007f321494b184 00007f321494b112 00007f321494b041 00007f321494afbc 00007f321494a272 00007f321474994c 00007f3214748d86
0806 03:43:48.229946 (+ 2861us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1172262784}
I20260806 03:43:48.144219 4482 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:48.245940 1623 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.022s sys 0.055s
I20260806 03:43:48.248409 4417 raft_consensus.cc:3032] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:48.248867 4000 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.627s user 0.010s sys 0.000s
W20260806 03:43:48.249658 558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1232) took 2373 ms (client timeout 1816 ms). Trace:
I20260806 03:43:48.185613 4326 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.250993 562 raft_consensus.cc:1261] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 03:43:48.251581 4446 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:48.256690 1623 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.022s sys 0.056s
I20260806 03:43:48.344092 4363 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:48.292816 558 rpcz_store.cc:269] 0806 03:43:45.875696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.875854 (+ 158us) service_pool.cc:227] Handling call
0806 03:43:48.249364 (+2373510us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b540030eb54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:48.249544 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2204179840}
W20260806 03:43:48.352737 4414 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.500s user 0.008s sys 0.002s
I20260806 03:43:48.266883 4417 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:48.525146 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 769222us
I20260806 03:43:48.442621 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:48.395669 3922 consensus_meta.cc:287] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.687s user 0.008s sys 0.004s
I20260806 03:43:48.356849 3907 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.546425 1622 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:48.547103 4482 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.547714 4363 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:48.548650 4427 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:48.581902 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:48.583693 1665 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 03:43:48.584110 1665 pending_rounds.cc:76] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting all ops after (but not including) 6
I20260806 03:43:48.584226 4427 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:48.584990 4427 raft_consensus.cc:686] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:48.586470 3905 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:48.551299 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.581925 4363 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.587715 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.588837 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.589406 1658 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:48.589562 4363 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.586097 4427 consensus_queue.cc:237] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.584291 1665 pending_rounds.cc:97] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 03:43:48.595782 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.596135 1665 raft_consensus.cc:2866] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:48.596431 776 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:48.598909 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:48.556078 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:48.552241 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2269) took 1357 ms. Trace:
I20260806 03:43:48.578739 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1277) took 1133 ms. Trace:
W20260806 03:43:48.550223 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:48.472086 4493 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.652s user 0.012s sys 0.000s
W20260806 03:43:48.606554 376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 56373.7us
I20260806 03:43:48.608616 775 rpcz_store.cc:276] 0806 03:43:47.195060 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.199622 (+ 4562us) service_pool.cc:227] Handling call
0806 03:43:47.199989 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:47.200236 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:43:47.251611 (+ 51375us) log.cc:842] Serialized 14 byte log entry
0806 03:43:47.901511 (+649900us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b18002e6ea0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa657b 00007f321632988a
0806 03:43:48.538986 (+637475us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:48.539068 (+ 82us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:48.539139 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:48.539230 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:48.552166 (+ 12936us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":48207,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":2677,"raft.queue_time_us":108,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":8816,"spinlock_wait_cycles":31883520,"thread_start_us":172320,"threads_started":1}
I20260806 03:43:48.612236 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.613554 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:48.618979 776 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:43:48.616586 4473 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.622015 1663 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 03:43:48.606155 4422 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:48.580358 4410 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:48.607296 547 rpcz_store.cc:276] 0806 03:43:47.445487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.696007 (+250520us) service_pool.cc:227] Handling call
0806 03:43:47.713221 (+ 17214us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:47.713472 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:43:47.777585 (+ 64113us) log.cc:842] Serialized 14 byte log entry
0806 03:43:48.578246 (+800661us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:48.578319 (+ 73us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:48.578410 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:48.578500 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:48.578668 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":800161,"prepare.queue_time_us":90703,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":809338,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94}
W20260806 03:43:48.620589 728 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.554s user 0.002s sys 0.029s
I20260806 03:43:48.575255 4476 consensus_queue.cc:260] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.610589 4354 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.613559 4363 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:48.628468 4417 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:48.647280 728 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.581s user 0.004s sys 0.029s
I20260806 03:43:48.618352 4547 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:48.654990 669 raft_consensus.cc:1226] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
W20260806 03:43:48.657776 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 592013us
I20260806 03:43:48.621932 4473 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.604790 4482 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.681342 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 132275us
I20260806 03:43:48.581151 3922 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.602358 3907 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.687207 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:48.690815 661 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:48.646247 4298 raft_consensus.cc:2741] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
W20260806 03:43:48.690898 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 134939us
I20260806 03:43:48.715996 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:48.647015 4350 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:48.681111 4114 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.009s sys 0.018s
I20260806 03:43:48.731002 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:48.646270 4510 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:48.614025 4493 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:48.726258 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 149267us
I20260806 03:43:48.722049 776 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:43:48.736369 4464 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.557s user 0.012s sys 0.002s
I20260806 03:43:48.738121 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.659075 4363 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.689723 4376 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.626852 4422 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.731172 4114 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.012s sys 0.020s
I20260806 03:43:48.623595 4523 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
I20260806 03:43:48.686265 4024 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.726649 4417 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:48.852588 4546 log.cc:924] Time spent T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20260806 03:43:48.853924 4473 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:48.854971 4530 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.117s user 0.000s sys 0.000s
I20260806 03:43:48.855285 4473 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.795320 4510 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:48.856094 4539 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260806 03:43:48.855875 4473 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.857522 4510 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:48.862779 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1276) took 1417 ms. Trace:
I20260806 03:43:48.863847 4363 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.864265 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 491) took 2754 ms. Trace:
I20260806 03:43:48.863216 546 rpcz_store.cc:276] 0806 03:43:47.445208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.695766 (+250558us) service_pool.cc:227] Handling call
0806 03:43:47.696157 (+ 391us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:48.178783 (+482626us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:48.178815 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:48.179256 (+ 441us) log.cc:842] Serialized 26 byte log entry
0806 03:43:48.636520 (+457264us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:48.636573 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:48.636667 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:48.862504 (+225837us) raft_consensus.cc:1671] finished
0806 03:43:48.862527 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:48.862681 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":897634,"raft.queue_time_us":486086,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":456798,"threads_started":1,"wal-append.queue_time_us":632150}
I20260806 03:43:48.741437 548 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:48.654554 4354 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.866321 4375 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.868088 4354 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.628005 4410 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.785348 4298 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:48.786116 4289 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:48.870927 4289 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.871582 4289 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.872105 4410 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.874526 4493 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.789151 4446 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:48.790048 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.844312 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.682269 4547 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.849114 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 812) took 1645 ms. Trace:
I20260806 03:43:48.737172 4350 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.850567 4472 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.864487 4363 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:48.864645 668 rpcz_store.cc:276] 0806 03:43:46.110499 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:46.111052 (+ 553us) service_pool.cc:227] Handling call
0806 03:43:48.863723 (+2752671us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321b264c6e 00007f321495b807 00007f321495b792 00007f321495b6c1 00007f321495b63c 00007f321495a8f2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:48.864166 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3034584320}
I20260806 03:43:48.877804 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.986418 776 raft_consensus.cc:2396] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:48.878713 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.808864 4464 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.858088 4510 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:48.680888 4498 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:48.880797 4024 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:48.840734 4376 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.869880 4298 raft_consensus.cc:686] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:48.889273 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.880846 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:48.892679 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 81362.4us
I20260806 03:43:49.002969 766 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:48.881112 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.005568 775 raft_consensus.cc:2396] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:49.003228 772 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:48.791682 4057 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:48.895831 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 55342.6us
I20260806 03:43:48.929530 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:48.935909 767 pending_rounds.cc:76] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 6
I20260806 03:43:49.022511 767 pending_rounds.cc:97] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260806 03:43:49.022902 767 raft_consensus.cc:2866] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:48.858247 4473 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.867175 4375 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:48.942555 1671 rpcz_store.cc:276] 0806 03:43:47.203693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.203823 (+ 130us) service_pool.cc:227] Handling call
0806 03:43:47.268823 (+ 65000us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:47.419775 (+150952us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:47.419808 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:47.419852 (+ 44us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:47.419884 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:48.848844 (+1428960us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54004f4cd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:48.848895 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:48.849046 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":335752064}
I20260806 03:43:48.849383 4422 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.950591 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1270) took 1748 ms. Trace:
I20260806 03:43:48.952227 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1388) took 1858 ms. Trace:
W20260806 03:43:48.952348 1670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 605) took 1788 ms (client timeout 1983 ms). Trace:
I20260806 03:43:48.959153 4556 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:48.960273 615 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:48.889858 4289 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:48.972455 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.980549 661 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:48.853991 4482 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:48.855295 4101 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:48.989059 614 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:48.898026 3922 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:48.940997 4546 log.cc:924] Time spent T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.087s user 0.003s sys 0.000s
I20260806 03:43:48.994238 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.843937 4461 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.138s
I20260806 03:43:48.944312 4411 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.011123 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.002952 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:48.995962 4363 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:49.024127 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 94718us
I20260806 03:43:49.024276 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:48.990295 4472 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:49.024005 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 308114us
I20260806 03:43:49.028694 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.211530 3883 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.025213 4057 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.214318 3883 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.216101 4057 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.216435 3883 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:49.204902 4061 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.280s user 0.015s sys 0.000s
I20260806 03:43:49.217902 4061 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.222970 4464 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:49.111757 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 299595us
W20260806 03:43:49.224457 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 195887us
I20260806 03:43:49.164512 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 364) took 4177 ms. Trace:
I20260806 03:43:49.029381 771 rpcz_store.cc:276] 0806 03:43:47.093575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.098489 (+ 4914us) service_pool.cc:227] Handling call
0806 03:43:48.951942 (+1853453us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b540030ff54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:48.952145 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803413248}
I20260806 03:43:49.052093 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.227211 776 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:49.096748 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 149916us
I20260806 03:43:49.097501 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:49.228121 1670 rpcz_store.cc:269] 0806 03:43:47.163736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.165146 (+ 1410us) service_pool.cc:227] Handling call
0806 03:43:48.952049 (+1786903us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54004af954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:48.952256 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":583184128}
I20260806 03:43:49.235605 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:49.236596 766 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:49.230482 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.029196 544 rpcz_store.cc:276] 0806 03:43:47.202606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.202723 (+ 117us) service_pool.cc:227] Handling call
0806 03:43:47.203071 (+ 348us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:48.125464 (+922393us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:48.125508 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:48.144352 (+ 18844us) log.cc:842] Serialized 52 byte log entry
0806 03:43:48.263530 (+119178us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:48.334662 (+ 71132us) log.cc:842] Serialized 14 byte log entry
0806 03:43:48.703982 (+369320us) log.cc:842] Serialized 14 byte log entry
0806 03:43:48.789912 (+ 85930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:48.949823 (+159911us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b540011f3d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:48.949905 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:48.950173 (+ 268us) raft_consensus.cc:1671] finished
0806 03:43:48.950197 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:48.950494 (+ 297us) inbound_call.cc:173] Queueing success response
0806 03:43:49.199845 (+249351us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b18003ec6e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
Metrics: {"mutex_wait_us":1373101,"prepare.queue_time_us":101120,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":7332,"raft.queue_time_us":1152699,"raft.run_cpu_time_us":590,"raft.run_wall_time_us":67805,"spinlock_wait_cycles":602024064,"thread_start_us":80762,"threads_started":1,"wal-append.queue_time_us":459223}
I20260806 03:43:49.231467 501 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:49.243731 727 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.225327 561 rpcz_store.cc:276] 0806 03:43:44.986708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:44.986847 (+ 139us) service_pool.cc:227] Handling call
0806 03:43:46.628049 (+1641202us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:46.652208 (+ 24159us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:49.112599 (+2460391us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b5400086450. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:49.164428 (+ 51829us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3773804416}
I20260806 03:43:49.029922 1663 raft_consensus.cc:2396] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:43:49.247479 544 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:49.245102 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.247942 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.248852 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:49.198973 4549 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 03:43:49.251413 546 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 03:43:49.251506 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:49.255649 558 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.251746 4289 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:49.255177 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 295037us
I20260806 03:43:49.253136 727 leader_election.cc:398] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 3. Current term is 4.
I20260806 03:43:49.254175 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:49.163009 4015 consensus_meta.cc:287] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.334s user 0.010s sys 0.005s
I20260806 03:43:49.257588 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.253913 4388 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.005s
I20260806 03:43:49.200994 4298 consensus_queue.cc:237] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.254077 4422 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:49.259709 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.260227 498 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.198840 4376 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.301993 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.302316 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:49.303002 564 raft_consensus.cc:2396] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 03:43:49.260876 4057 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:49.262231 3883 consensus_queue.cc:1046] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:49.304666 3883 raft_consensus.cc:3032] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:49.304654 4567 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:49.305572 3883 raft_consensus.cc:729] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:49.288060 4553 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:49.263173 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.306684 1663 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:49.306494 3883 consensus_queue.cc:260] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.263907 4547 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:49.263691 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 278824us
I20260806 03:43:49.308527 4024 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.254732 4447 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:49.309484 614 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.270884 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.050846 4443 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:43:49.310853 4482 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:49.311430 4443 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.311815 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.312132 4443 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:43:49.312783 1657 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:49.309425 4447 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.313704 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.272236 771 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:49.243479 4411 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.242151 4101 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.100760 4560 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.338573 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.363091 564 raft_consensus.cc:2396] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:43:49.323926 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.278435 4550 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.371409 558 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:49.373989 776 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:49.374354 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.389569 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.382053 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.398824 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:49.316021 4564 log.cc:924] Time spent T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.105s user 0.001s sys 0.001s
I20260806 03:43:49.002769 4498 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.274552 616 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.296365 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.325976 4565 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:43:49.299614 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46680.3us
I20260806 03:43:49.301156 3905 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:49.301229 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.306166 4553 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:49.333591 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.276717 4523 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.311602 4482 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.443197 4482 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.442974 4523 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.444753 1671 raft_consensus.cc:3032] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:49.445225 1671 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:49.275823 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:49.245733 4555 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
I20260806 03:43:49.446686 544 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
W20260806 03:43:49.266492 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 333431us
I20260806 03:43:49.449513 3883 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:49.400575 558 raft_consensus.cc:2362] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7. Re-sending same reply.
I20260806 03:43:49.451169 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.452505 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.453820 558 raft_consensus.cc:1226] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:49.433593 4553 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.314329 4447 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:49.457065 3907 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:49.457140 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 171578us
I20260806 03:43:49.297757 4554 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.409636 547 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:49.412986 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.482897 564 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:49.324213 4350 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.323238 4375 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.319404 4098 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:49.492419 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:49.423025 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 148554us
I20260806 03:43:49.423579 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.320187 4472 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:49.496811 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 261281us
I20260806 03:43:49.426362 727 leader_election.cc:398] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:49.242923 4556 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.322507 4363 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.326887 4422 raft_consensus.cc:686] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:49.431627 1670 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:49.321415 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:49.398093 4478 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.480s user 0.012s sys 0.019s
I20260806 03:43:49.439376 4498 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.451072 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.409610 561 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:49.012147 4510 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.463105 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.463155 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.458681 615 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:49.446152 1671 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.292268 4015 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.497726 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:49.433125 4550 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:49.435469 4464 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260806 03:43:49.505564 616 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:49.431171 4565 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.506635 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.507490 547 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:49.433892 4289 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:49.517666 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 59131.2us
I20260806 03:43:49.504108 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:49.513667 4478 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.596s user 0.012s sys 0.020s
I20260806 03:43:49.544909 616 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.537617 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.537998 771 raft_consensus.cc:2396] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:49.531114 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.528641 542 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:49.547793 776 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.551571 766 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:49.555428 1671 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.527281 4363 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:49.533488 4550 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:49.560170 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 134006us
I20260806 03:43:49.565196 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.565841 766 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:49.549324 4417 consensus_meta.cc:287] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.698s user 0.011s sys 0.011s
I20260806 03:43:49.569276 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.569831 4101 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:49.539252 4565 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:49.580940 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.570051 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.601230 773 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:49.601466 772 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:49.577584 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.578147 4567 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:49.578389 4582 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:49.578686 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.603586 776 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:49.579151 4477 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:49.579057 4575 consensus_queue.cc:1035] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
W20260806 03:43:49.579704 4571 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.054s user 0.002s sys 0.000s
I20260806 03:43:49.619838 769 raft_consensus.cc:2396] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:49.579787 4356 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.580379 4024 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:49.581041 4572 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260806 03:43:49.378017 4560 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.582335 4586 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:49.586066 4562 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.586853 4574 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:49.599953 4422 consensus_queue.cc:237] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.666807 4574 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:49.667563 4574 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.588178 4015 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.590029 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:49.671327 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 81385.8us
I20260806 03:43:49.595502 727 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:49.570667 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 78316.2us
I20260806 03:43:49.602701 771 raft_consensus.cc:2396] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:43:49.674304 4417 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.674815 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.675194 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.675932 771 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:49.675657 4562 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.677002 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:49.598271 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 60789.2us
W20260806 03:43:49.559412 1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 815) took 1190 ms (client timeout 1521 ms). Trace:
I20260806 03:43:49.540853 4478 raft_consensus.cc:686] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:49.678593 1660 rpcz_store.cc:269] 0806 03:43:48.368670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.320896 (+952226us) service_pool.cc:227] Handling call
0806 03:43:49.555833 (+234937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:49.679373 4556 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.612820 1671 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:49.680600 771 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:49.492914 4554 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:49.690847 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 141859us
I20260806 03:43:49.578148 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.552016 4289 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.657243 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.614357 4298 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.611060 4423 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.665896 4000 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:49.673549 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 107866us
I20260806 03:43:49.719275 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.677302 4473 raft_consensus.cc:3032] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:49.681360 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.719962 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.684366 501 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.610886 4567 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.570892 4550 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:49.626636 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.599776 4523 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:49.653954 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 135312us
I20260806 03:43:49.683183 4586 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:49.674333 4510 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.698438 772 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:49.699774 615 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.703226 669 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:49.722069 501 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:49.704149 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.736176 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.708562 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.715317 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.736953 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.680301 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.574970 4570 consensus_queue.cc:1046] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.164s
I20260806 03:43:49.739395 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.740165 665 raft_consensus.cc:2396] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:43:49.740025 501 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:49.611106 4582 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.720561 1671 raft_consensus.cc:2379] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 4: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:49.692513 4417 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:49.730784 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.738052 771 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:49.731407 1658 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:49.731926 671 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:49.610719 4553 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.733680 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.691162 4464 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.739919 615 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.735195 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.760066 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:49.770735 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.665982 4356 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.771442 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.765972 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.760061 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.663098 4024 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.768461 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.691411 4472 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:49.701718 4411 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.772518 772 raft_consensus.cc:2396] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:43:49.766542 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:49.759801 4513 log.cc:924] Time spent T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 1.990s user 0.017s sys 0.009s
I20260806 03:43:49.778851 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.806816 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.762109 3905 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:49.748153 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.786787 547 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:49.814122 773 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:49.725080 4567 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.731599 3883 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:49.724678 4473 raft_consensus.cc:729] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:49.698514 4497 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260806 03:43:49.794953 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.719975 4578 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:49.789438 4015 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.729979 4000 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.725050 4569 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:49.732569 4586 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.813028 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.682303 4447 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.813099 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.807108 665 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:49.742950 4562 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.820096 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:49.851163 669 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:49.829691 4443 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.517s user 0.013s sys 0.011s
I20260806 03:43:49.856391 1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 801) took 4206 ms. Trace:
I20260806 03:43:49.680207 4478 consensus_queue.cc:237] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:49.822530 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:49.767231 4326 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:49.740365 4592 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:49.759277 4582 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.813903 3905 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.828377 4289 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:49.829651 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.830572 4473 consensus_queue.cc:260] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.841774 4577 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:43:49.831830 4554 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:49.819602 4570 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.007s
I20260806 03:43:49.834437 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.779908 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.832924 4591 raft_consensus.cc:3032] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:49.833200 4553 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.840945 4528 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:49.841641 4588 consensus_queue.cc:1046] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260806 03:43:49.820370 4510 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.714355 4298 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:49.819407 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 100268us
I20260806 03:43:49.853855 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:49.821177 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.856416 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:49.865092 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.785912 4417 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.869314 547 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:49.855965 4497 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.864984 4326 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.873814 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.856971 1668 rpcz_store.cc:276] 0806 03:43:45.641634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.642606 (+ 972us) service_pool.cc:227] Handling call
0806 03:43:45.658301 (+ 15695us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:47.170152 (+1511851us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:47.170181 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:47.183411 (+ 13230us) log.cc:842] Serialized 26 byte log entry
0806 03:43:47.437903 (+254492us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:47.437963 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:47.438072 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:49.594935 (+2156863us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:49.844589 (+249654us) raft_consensus.cc:1671] finished
0806 03:43:49.844610 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:49.848343 (+ 3733us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1511448,"raft.queue_time_us":1534242,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":34922,"spinlock_wait_cycles":13837184,"thread_start_us":65206,"threads_started":1,"wal-append.queue_time_us":485835}
I20260806 03:43:49.873412 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.907421 669 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:49.879202 561 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 6. Ops: []
I20260806 03:43:49.723933 4423 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.810478 4472 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:43:49.833231 4578 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:49.887421 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53104.9us
I20260806 03:43:49.915477 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.917445 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.887452 498 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.888906 766 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:49.867527 4529 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:49.923760 4529 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.924408 4529 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.924621 4000 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.891109 663 raft_consensus.cc:2379] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:49.891227 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.926913 4577 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.928491 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.895860 4528 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:49.893381 4592 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:49.867712 4482 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.891544 4591 raft_consensus.cc:729] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:49.908588 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.926997 4298 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:49.907075 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 53301.1us
I20260806 03:43:49.880757 3905 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.893301 4473 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:49.867283 4443 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.918754 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.918797 498 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.944939 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.923027 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.902732 665 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
I20260806 03:43:49.945900 1668 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:49.947536 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:49.909444 727 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:49.934798 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.950037 498 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:49.931818 1663 consensus_meta.cc:287] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.607s user 0.015s sys 0.005s
I20260806 03:43:49.933139 4445 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:49.937517 4447 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:49.965184 1624 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:49.945690 669 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:49.910405 4326 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:49.953385 498 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.957738 499 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:49.965173 4605 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.967290 668 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:49.926981 4472 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:49.971024 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.971170 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:49.900525 4553 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.904459 4510 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:49.982645 558 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
W20260806 03:43:49.981966 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 66636.7us
I20260806 03:43:49.970932 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.935719 4015 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:49.946290 4523 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:49.969592 4298 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.984766 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.989427 1622 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:49.990238 669 raft_consensus.cc:1226] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:49.945850 4560 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.933951 4528 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.914111 4599 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:49.967273 4057 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:49.938901 4591 consensus_queue.cc:260] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.000380 1624 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:49.991824 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:49.982510 3883 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:49.985131 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:49.994316 4445 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:50.004379 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:50.020629 4445 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:50.020903 547 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.021242 4445 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:49.961236 1663 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:50.001761 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.980895 4318 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.013099 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.026836 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.013953 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.968225 4592 raft_consensus.cc:686] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:49.994468 4356 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.928414 4577 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:50.018026 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 115758us
I20260806 03:43:50.052569 727 leader_election.cc:398] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:43:50.052945 727 leader_election.cc:401] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:50.019949 4563 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:49.998868 4024 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:49.992977 4015 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:50.023869 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:49.913450 4497 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.010078 4554 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:43:50.070358 674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2346) took 3182 ms (client timeout 2205 ms). Trace:
W20260806 03:43:50.070981 674 rpcz_store.cc:269] 0806 03:43:46.887847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:46.888016 (+ 169us) service_pool.cc:227] Handling call
0806 03:43:50.067593 (+3179577us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540019b9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.070243 (+ 2650us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":489236352}
I20260806 03:43:50.030956 4603 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.030874 4560 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:50.031215 4443 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.032550 4417 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.033555 4578 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:50.028577 1660 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:50.086104 4423 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.087121 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:50.088089 674 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:43:50.088549 4577 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.089028 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:50.035241 4482 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.090927 4497 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.091132 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.039705 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 425) took 1179 ms. Trace:
W20260806 03:43:50.040627 548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1241) took 4132 ms (client timeout 2737 ms). Trace:
I20260806 03:43:50.092803 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:50.093012 548 rpcz_store.cc:269] 0806 03:43:45.908004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:45.939586 (+ 31582us) service_pool.cc:227] Handling call
0806 03:43:50.040396 (+4100810us) spinlock_profiling.cc:243] Waited 2.8 s on lock 0x7b540032d2d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.040543 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6440712960}
I20260806 03:43:49.993911 4553 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.057440 4289 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
W20260806 03:43:50.062270 770 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1385) took 2969 ms (client timeout 2242 ms). Trace:
I20260806 03:43:50.066715 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:50.069943 659 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:50.001911 4523 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.097563 773 raft_consensus.cc:3037] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:50.097287 770 rpcz_store.cc:269] 0806 03:43:47.092543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.094807 (+ 2264us) service_pool.cc:227] Handling call
0806 03:43:50.061993 (+2967186us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b5400126954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.062174 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1868217728}
I20260806 03:43:50.101912 4553 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.103080 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 820) took 1419 ms. Trace:
I20260806 03:43:50.102766 4445 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.103724 1662 rpcz_store.cc:276] 0806 03:43:48.683537 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:48.686729 (+ 3192us) service_pool.cc:227] Handling call
0806 03:43:50.101979 (+1415250us) spinlock_profiling.cc:243] Waited 5.21 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:50.102267 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.102517 (+ 250us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:50.102555 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:50.102627 (+ 72us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:50.102667 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:50.102781 (+ 114us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:50.102980 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11973888}
I20260806 03:43:50.103016 669 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260806 03:43:50.076094 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 93891.8us
I20260806 03:43:50.061333 4015 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:50.081820 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 421) took 1222 ms. Trace:
I20260806 03:43:50.061612 4563 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.084885 4061 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.080992 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.038702 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1267) took 3161 ms. Trace:
I20260806 03:43:50.009774 4599 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:50.113572 562 rpcz_store.cc:276] 0806 03:43:46.877038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:46.877597 (+ 559us) service_pool.cc:227] Handling call
0806 03:43:48.250504 (+1372907us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:48.297728 (+ 47224us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:50.038417 (+1740689us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540030eb50. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.038627 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3113495808}
I20260806 03:43:50.092564 1658 rpcz_store.cc:276] 0806 03:43:48.860242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.240187 (+379945us) service_pool.cc:227] Handling call
0806 03:43:50.039511 (+799324us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54004f4cd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.039635 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037458176}
I20260806 03:43:50.055727 4558 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.095376 728 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:50.111007 1670 rpcz_store.cc:276] 0806 03:43:48.858938 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.248617 (+389679us) service_pool.cc:227] Handling call
0806 03:43:50.081638 (+833021us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b5400579254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.081748 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":418683648}
I20260806 03:43:50.121418 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.112200 727 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:50.118096 615 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:50.123131 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:50.125859 1668 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:50.127436 4578 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:50.128311 4578 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:50.129040 4510 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.130633 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.132148 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.133404 4523 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.129052 4578 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:50.135205 667 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:50.119290 4445 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.145569 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.115957 4423 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:50.146322 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.146474 1658 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:50.152050 728 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:50.120079 4289 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.120230 4599 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:50.157691 674 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:50.126238 1663 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:50.143992 4510 raft_consensus.cc:2741] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 03:43:50.138847 4577 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:50.177321 4288 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:43:50.192876 728 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:50.194556 674 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:43:50.164253 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.198153 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:50.199062 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.199855 663 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:50.192768 4288 raft_consensus.cc:686] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:50.202212 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:50.203433 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.183965 4423 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:50.203924 776 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:50.199950 772 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:50.204030 674 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 1.
I20260806 03:43:50.177783 4577 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:50.208295 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:50.210829 1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 606) took 3046 ms (client timeout 1725 ms). Trace:
I20260806 03:43:50.209542 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:50.211216 1664 rpcz_store.cc:269] 0806 03:43:47.164022 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.164156 (+ 134us) service_pool.cc:227] Handling call
0806 03:43:50.210648 (+3046492us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400515754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.210761 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":489978624}
I20260806 03:43:50.210029 1658 raft_consensus.cc:3032] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:50.212101 1658 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:50.217446 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:50.124258 4607 consensus_queue.cc:1159] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" responder_term: 6 status { last_received { term: 6 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 6 index: 6 } }
I20260806 03:43:50.226624 776 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:50.221673 773 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:50.220824 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.235611 663 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:50.236156 1624 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:50.237051 727 connection.cc:232] RPC call timeout handler was delayed by -2.8314326740801334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
W20260806 03:43:50.238865 727 connection.cc:232] RPC call timeout handler was delayed by -2.5956326741725206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260806 03:43:50.209468 4288 consensus_queue.cc:237] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.245251 766 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260806 03:43:50.245820 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 934682us
I20260806 03:43:50.210428 4577 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:50.150266 4569 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.062s
I20260806 03:43:50.219808 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:50.239217 727 connection.cc:232] RPC call timeout handler was delayed by -2.5722326741088182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
W20260806 03:43:50.254081 727 connection.cc:232] RPC call timeout handler was delayed by -2.3514232700690627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20260806 03:43:50.254472 727 connection.cc:232] RPC call timeout handler was delayed by -2.3490326742175967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
I20260806 03:43:50.255033 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.254070 615 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.235867 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:50.248740 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.251287 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.140064 4592 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:50.259548 4610 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.138s user 0.000s sys 0.002s
I20260806 03:43:50.216517 4510 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:50.219918 4318 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:50.211328 4423 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:50.262791 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 273461us
I20260806 03:43:50.237006 1658 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.188843 4529 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.268186 4510 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:50.236747 4417 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:50.272732 1658 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.272893 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.276715 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:50.267663 4565 consensus_meta.cc:287] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.639s user 0.003s sys 0.011s
I20260806 03:43:50.289546 1622 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.290694 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.293326 663 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:50.294621 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.296339 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:50.262017 4560 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.297257 663 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:50.267243 4558 raft_consensus.cc:2781] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:50.317314 1658 raft_consensus.cc:1261] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:50.323258 547 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:50.275231 4510 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.267959 4318 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
W20260806 03:43:50.381560 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 71851.3us
I20260806 03:43:50.385063 614 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:50.279525 4423 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:50.304793 4574 consensus_meta.cc:287] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.637s user 0.014s sys 0.004s
I20260806 03:43:50.278746 4603 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:43:50.388456 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 57683.4us
I20260806 03:43:50.391340 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.309643 4558 raft_consensus.cc:686] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:50.300855 4529 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:50.396698 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.396797 4529 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:50.396579 4558 consensus_queue.cc:237] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.397913 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.394250 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:50.304963 4620 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:50.400844 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 208049us
I20260806 03:43:50.300922 4356 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260806 03:43:50.287324 4482 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.108s
I20260806 03:43:50.417976 614 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:50.308132 4554 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:50.429003 663 raft_consensus.cc:1261] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:43:50.432282 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:50.433894 776 raft_consensus.cc:1261] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 03:43:50.389690 4586 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.529s user 0.006s sys 0.007s
I20260806 03:43:50.388304 4318 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.440182 4627 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:50.449429 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:50.401597 1660 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:43:50.484031 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 51878.6us
I20260806 03:43:50.484081 4554 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:50.430058 4425 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:50.410775 4611 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:50.312906 4565 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.454352 4482 consensus_queue.cc:1225] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer ada5ef6656cb4420baf77a4b4342909e log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:43:50.482220 4374 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:50.535422 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 86145.9us
W20260806 03:43:50.543290 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:50.515265 4288 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:43:50.409307 4490 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.116s
I20260806 03:43:50.519939 4554 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:50.516602 4356 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.402484 4574 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:50.627430 4619 log.cc:924] Time spent T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20260806 03:43:50.630645 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2352) took 3200 ms. Trace:
I20260806 03:43:50.637917 664 rpcz_store.cc:276] 0806 03:43:47.430283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.430615 (+ 332us) service_pool.cc:227] Handling call
0806 03:43:50.630428 (+3199813us) spinlock_profiling.cc:243] Waited 60.7 ms on lock 0x7b540019afd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.630576 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139723648}
I20260806 03:43:50.547676 4483 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:43:50.638859 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 139665us
W20260806 03:43:50.649680 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 687970us
I20260806 03:43:50.501662 4586 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.650638 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.655160 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43080 (request call id 19) took 3472 ms. Trace:
I20260806 03:43:50.656140 4586 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.655714 563 rpcz_store.cc:276] 0806 03:43:47.182282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:47.184274 (+ 1992us) service_pool.cc:227] Handling call
0806 03:43:50.654823 (+3470549us) spinlock_profiling.cc:243] Waited 30.7 ms on lock 0x7b540030e654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.654979 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70627584}
I20260806 03:43:50.656958 4553 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260806 03:43:50.660058 4298 consensus_meta.cc:287] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.656s user 0.009s sys 0.004s
I20260806 03:43:50.660645 4298 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.661542 775 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:50.663637 4528 consensus_meta.cc:287] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.649s user 0.013s sys 0.001s
I20260806 03:43:50.664440 4528 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.669878 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:50.660087 4476 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:50.641535 4483 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:50.654305 4326 consensus_meta.cc:287] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.659s user 0.007s sys 0.006s
I20260806 03:43:50.677683 4576 raft_consensus.cc:3032] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:50.712141 4576 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:50.634191 4130 consensus_queue.cc:1035] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.051s
W20260806 03:43:50.715145 3905 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.748s user 0.005s sys 0.009s
W20260806 03:43:50.714977 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 69788.8us
I20260806 03:43:50.716226 3905 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.670704 4268 consensus_queue.cc:1046] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260806 03:43:50.739122 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2289) took 1233 ms. Trace:
I20260806 03:43:50.740229 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2369) took 1115 ms. Trace:
I20260806 03:43:50.741302 767 raft_consensus.cc:1261] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:50.670037 4566 raft_consensus.cc:3032] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:50.740955 667 rpcz_store.cc:276] 0806 03:43:49.628885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.674497 (+ 45612us) service_pool.cc:227] Handling call
0806 03:43:50.739994 (+1065497us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b540019bed4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.740143 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1057017600}
W20260806 03:43:50.723284 4582 consensus_meta.cc:287] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.838s user 0.009s sys 0.009s
I20260806 03:43:50.441480 4620 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.739923 769 rpcz_store.cc:276] 0806 03:43:49.506225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.506349 (+ 124us) service_pool.cc:227] Handling call
0806 03:43:50.738755 (+1232406us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b5400320854. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.739018 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189764992}
I20260806 03:43:50.705098 1662 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:50.704370 4483 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.528882 4627 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.772382 1664 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
W20260806 03:43:50.773523 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 484065us
I20260806 03:43:50.782305 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.784369 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:50.785470 775 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 5
W20260806 03:43:50.766820 4550 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.995s user 0.011s sys 0.007s
W20260806 03:43:50.775074 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.010s sys 0.019s
I20260806 03:43:50.667176 4599 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.685088 4586 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
W20260806 03:43:50.794296 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s user 0.011s sys 0.020s
W20260806 03:43:50.795625 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 540624us
I20260806 03:43:50.768167 4566 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:50.745064 4576 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.726432 4326 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.809751 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.811993 4130 consensus_queue.cc:1046] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:50.817700 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.801309 4114 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:50.771658 4431 consensus_queue.cc:1035] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260806 03:43:50.827694 4636 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:50.810956 4528 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.822585 4298 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:50.822482 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:50.829699 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:50.769536 4582 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.827899 3905 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.841688 663 raft_consensus.cc:2379] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:50.830979 727 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:50.832657 4567 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.989s user 0.007s sys 0.010s
I20260806 03:43:50.826431 4368 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:50.849429 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.835754 4576 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:50.841990 4473 consensus_meta.cc:287] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.890s user 0.009s sys 0.007s
I20260806 03:43:50.853286 4567 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.859050 4627 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:50.799810 4550 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.860039 4602 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260806 03:43:50.867101 661 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:50.865831 667 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:50.868738 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.869247 769 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:50.854611 4368 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:50.872916 727 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:50.852135 4574 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:50.874203 4298 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:50.844882 4566 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:50.874854 4298 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:50.875550 4298 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:50.877803 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.879201 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:50.881215 671 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:50.813524 4640 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.060s
I20260806 03:43:50.884703 773 raft_consensus.cc:1261] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:50.880533 4560 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20260806 03:43:50.892489 665 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:50.894719 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.898555 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:50.940514 667 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:50.900099 775 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:50.883034 4627 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:43:50.930778 668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 359) took 1187 ms (client timeout 1511 ms). Trace:
I20260806 03:43:50.936734 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:50.941442 668 rpcz_store.cc:269] 0806 03:43:49.742951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.786417 (+ 43466us) service_pool.cc:227] Handling call
0806 03:43:50.930496 (+1144079us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540015a9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:50.930679 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258488320}
I20260806 03:43:50.941864 616 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.918229 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.942232 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:50.896674 4288 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:43:50.901463 4289 consensus_meta.cc:287] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.743s user 0.017s sys 0.000s
W20260806 03:43:50.928241 4643 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.061s user 0.001s sys 0.001s
I20260806 03:43:50.983271 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:50.894652 4560 consensus_queue.cc:1035] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.091s
I20260806 03:43:50.987762 4575 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:50.904567 4569 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:51.018589 4569 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:51.019273 4569 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:50.899595 4620 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:50.930801 4567 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.927016 4621 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:50.994565 4289 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:50.897439 4574 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.084s
W20260806 03:43:51.040930 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 1.04357e+06us
I20260806 03:43:50.980329 4482 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:51.043491 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2295) took 1456 ms. Trace:
I20260806 03:43:51.044021 768 rpcz_store.cc:276] 0806 03:43:49.586796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.781253 (+194457us) service_pool.cc:227] Handling call
0806 03:43:49.781568 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:49.950842 (+169274us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:49.950884 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:49.950930 (+ 46us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:49.950978 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.043195 (+1092217us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54002bded4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:51.043254 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:51.043420 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":388761856}
I20260806 03:43:50.992755 4318 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:51.013230 4645 log.cc:924] Time spent T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260806 03:43:50.946713 4636 raft_consensus.cc:2741] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:43:50.947768 4627 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.015492 4368 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:50.979297 4326 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.060521 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:50.981657 4288 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:51.016888 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:50.960237 4602 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:50.940917 4599 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.069499 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 376) took 1582 ms. Trace:
I20260806 03:43:51.069840 4550 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.071314 775 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:50.935624 4640 consensus_queue.cc:1035] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260806 03:43:51.042793 4482 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:51.083475 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 432923us
I20260806 03:43:51.084064 4114 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:51.085561 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:50.980014 4431 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:43:51.048161 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 145076us
W20260806 03:43:51.029832 4423 consensus_meta.cc:287] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.627s user 0.019s sys 0.000s
I20260806 03:43:51.057879 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:51.057546 4627 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:51.063320 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:51.090981 768 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:51.053530 4015 consensus_meta.cc:287] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.933s user 0.011s sys 0.005s
I20260806 03:43:51.067994 770 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:51.070053 564 rpcz_store.cc:276] 0806 03:43:49.486749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.486874 (+ 125us) service_pool.cc:227] Handling call
0806 03:43:49.492358 (+ 5484us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:49.492678 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:49.492704 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:49.493151 (+ 447us) log.cc:842] Serialized 26 byte log entry
0806 03:43:50.628489 (+1135338us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18001b0fa0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa6411 00007f32162f35cf
0806 03:43:50.793867 (+165378us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:50.793950 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:50.794078 (+ 128us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:51.061811 (+267733us) raft_consensus.cc:1671] finished
0806 03:43:51.061835 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:51.069411 (+ 7576us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3792,"raft.queue_time_us":171,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":10713,"spinlock_wait_cycles":3496832,"thread_start_us":165051,"threads_started":1,"wal-append.queue_time_us":1328931}
I20260806 03:43:51.080295 4000 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:51.080807 4498 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:51.073012 4288 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:51.082513 4443 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:51.095551 616 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:51.063896 4582 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:51.081060 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.08924e+06us
I20260806 03:43:51.109887 776 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:51.098214 3906 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:51.092331 3905 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:51.105834 668 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260806 03:43:51.108297 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:51.108989 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:51.122925 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:51.089663 4482 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.093287 4550 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.098814 4431 raft_consensus.cc:686] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:51.106622 4595 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 1.266s user 0.003s sys 0.005s
I20260806 03:43:51.133816 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:51.099114 4469 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.140537 1658 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:51.102304 4498 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:51.168040 4498 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:51.168959 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 65431us
I20260806 03:43:51.151546 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.171952 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:51.123819 4577 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.873s user 0.016s sys 0.004s
I20260806 03:43:51.159231 770 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:51.164176 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.108505 4298 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.165206 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.167069 4318 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.117496 4443 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.155251 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:43:51.146761 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.153878 3814 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:51.209537 3814 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.142140 4567 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:51.210855 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.210948 4472 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:51.211381 4567 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:51.212175 4567 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:51.212080 4472 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.213287 3814 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.170877 4350 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:51.215889 4649 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260806 03:43:51.217545 563 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:51.183997 4582 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 03:43:51.182287 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.178068 4469 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.220121 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.220966 562 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:51.222025 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.222839 664 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:51.076781 4602 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.205822 3357 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:51.153525 4461 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:51.210242 4024 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:51.157519 4550 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.129179 4574 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260806 03:43:51.146139 4621 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260806 03:43:51.098995 4423 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.230574 1664 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
I20260806 03:43:51.103684 4015 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.270081 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.272188 546 raft_consensus.cc:2396] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:51.240481 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.238492 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.277631 564 raft_consensus.cc:2396] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:51.282354 563 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:51.287050 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:51.289121 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.290438 667 raft_consensus.cc:2379] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:51.259603 548 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:51.294025 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.297935 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:51.276194 4562 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.380s user 0.024s sys 0.021s
I20260806 03:43:51.156047 4482 consensus_queue.cc:1035] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.126s
I20260806 03:43:51.303941 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.305572 562 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:51.291546 4510 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.899s user 0.013s sys 0.006s
I20260806 03:43:51.304265 548 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:51.319483 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.324326 770 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:51.158555 4626 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.124s
I20260806 03:43:51.326094 616 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:51.328212 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:51.271497 4469 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.227078 3357 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:51.307296 4562 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.414s user 0.024s sys 0.021s
I20260806 03:43:51.343546 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.281306 4461 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.188795 4577 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.356361 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.357010 548 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:51.358920 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 189030us
I20260806 03:43:51.316267 4640 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.316128 4650 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.353212 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.281076 4024 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.331918 3906 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:51.333196 4510 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.543040 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 822) took 2856 ms. Trace:
I20260806 03:43:51.544631 1657 rpcz_store.cc:276] 0806 03:43:48.686624 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.410042 (+723418us) service_pool.cc:227] Handling call
0806 03:43:49.410421 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:49.425986 (+ 15565us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:49.426016 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:49.426055 (+ 39us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:49.426088 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.542753 (+2116665us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b54005e3654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:51.542815 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:51.542963 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35237632}
I20260806 03:43:51.545965 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:51.303037 4431 consensus_queue.cc:237] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.545987 4298 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.540642 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 375) took 2171 ms. Trace:
I20260806 03:43:51.364207 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.549470 558 rpcz_store.cc:276] 0806 03:43:49.384405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.467259 (+ 82854us) service_pool.cc:227] Handling call
0806 03:43:49.505564 (+ 38305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:49.987526 (+481962us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.540318 (+1552792us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400390154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:51.540556 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1096776576}
I20260806 03:43:51.549901 564 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:51.542853 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:51.552059 664 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:51.487663 4015 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.547190 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:51.553330 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:51.560258 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:51.498586 768 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:51.364609 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:51.567963 498 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:51.568246 776 raft_consensus.cc:1226] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:51.568725 770 raft_consensus.cc:2379] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:51.570163 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.336251 4560 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:51.440568 548 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
W20260806 03:43:51.542119 4578 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.404s user 0.018s sys 0.001s
I20260806 03:43:51.338989 3905 raft_consensus.cc:686] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:51.536612 4657 log.cc:924] Time spent T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.257s user 0.002s sys 0.000s
I20260806 03:43:51.576221 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:51.577713 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.593770 768 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:51.594758 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.436944 4152 raft_consensus.cc:3032] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:51.365763 4620 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.564163 4443 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.583316 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.586891 772 raft_consensus.cc:1226] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:51.589175 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:51.363538 4626 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:51.595719 548 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:51.608325 564 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:51.442759 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:51.397980 4472 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.488189 4461 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.578727 498 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:51.585741 4350 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:51.577124 616 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:51.603226 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:51.576066 3357 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.615793 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:51.489068 4423 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.635460 548 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
W20260806 03:43:51.641271 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 62675.6us
I20260806 03:43:51.643378 667 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:51.589227 4015 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:51.646394 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:51.647114 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 70394.8us
W20260806 03:43:51.647789 615 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.953s (SENT) [suppressed 11 similar messages]
I20260806 03:43:51.378816 4640 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.597726 4152 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:51.605031 4602 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:51.568970 4650 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.670047 548 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:51.596181 4578 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.508271 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:51.851910 768 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:52.509039 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:51.833735 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 95149.9us
W20260806 03:43:52.509454 768 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2312) took 1382 ms (client timeout 1758 ms). Trace:
I20260806 03:43:52.509800 770 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 4.
W20260806 03:43:52.509922 768 rpcz_store.cc:269] 0806 03:43:51.127197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.575968 (+448771us) service_pool.cc:227] Handling call
0806 03:43:52.509394 (+933426us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:52.510499 770 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2315) took 1379 ms (client timeout 1790 ms). Trace:
I20260806 03:43:52.511037 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.509151 1657 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:52.512163 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2309) took 1386 ms. Trace:
I20260806 03:43:52.512532 768 rpcz_store.cc:276] 0806 03:43:51.126087 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.510812 (+1384725us) service_pool.cc:227] Handling call
0806 03:43:52.512108 (+ 1296us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:51.794441 1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 414) took 3209 ms. Trace:
I20260806 03:43:52.121415 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:43:52.513163 564 consensus_meta.cc:287] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.886s user 0.026s sys 0.016s
I20260806 03:43:51.865209 667 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
W20260806 03:43:51.937732 671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2383) took 1946 ms (client timeout 1653 ms). Trace:
W20260806 03:43:52.517650 671 rpcz_store.cc:269] 0806 03:43:49.991709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.995281 (+ 3572us) service_pool.cc:227] Handling call
0806 03:43:51.937363 (+1942082us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b540036ced4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:51.937612 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1961010048}
W20260806 03:43:51.954663 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 306901us
I20260806 03:43:52.250945 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 839) took 1203 ms. Trace:
W20260806 03:43:52.340179 727 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.004s sys 0.009s
W20260806 03:43:52.520195 727 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.766s user 0.004s sys 0.014s
I20260806 03:43:52.519809 1664 rpcz_store.cc:276] 0806 03:43:51.047374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.047521 (+ 147us) service_pool.cc:227] Handling call
0806 03:43:52.250684 (+1203163us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54004f0454. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:52.250852 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":388796672}
W20260806 03:43:52.520526 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 766945us
I20260806 03:43:52.119422 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 535) took 2420 ms. Trace:
I20260806 03:43:52.245062 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 378) took 2755 ms. Trace:
W20260806 03:43:52.177326 1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 433) took 2260 ms (client timeout 1953 ms). Trace:
W20260806 03:43:52.527832 1668 rpcz_store.cc:269] 0806 03:43:49.916795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.997742 (+ 80947us) service_pool.cc:227] Handling call
0806 03:43:52.177001 (+2179259us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b54004af954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:52.177207 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227763968}
I20260806 03:43:52.514168 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:52.529388 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1297) took 1175 ms. Trace:
I20260806 03:43:51.624747 4550 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.515398 1665 rpcz_store.cc:276] 0806 03:43:48.585030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:48.597280 (+ 12250us) service_pool.cc:227] Handling call
0806 03:43:50.141285 (+1544005us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b5800005498. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2752b0 00007f321b2607f9 00007f321b2602a2 00007f3214946684
0806 03:43:50.175802 (+ 34517us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.176020 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:50.176052 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:50.176094 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:50.176131 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:50.176187 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:51.794136 (+1617949us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b54004f5e50. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:51.794333 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":126810368}
W20260806 03:43:51.734659 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 90029.1us
I20260806 03:43:52.525242 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:52.113584 4476 consensus_meta.cc:287] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.430s user 0.018s sys 0.002s
I20260806 03:43:51.672003 3906 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:51.793423 3905 consensus_queue.cc:237] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.535665 4130 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.536223 4130 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.536906 4578 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.537770 4578 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:51.847898 4623 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 1.317s user 0.001s sys 0.003s
W20260806 03:43:52.510972 770 rpcz_store.cc:269] 0806 03:43:51.131710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.653763 (+522053us) service_pool.cc:227] Handling call
0806 03:43:52.510437 (+856674us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:51.655689 4653 log.cc:924] Time spent T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.304s user 0.003s sys 0.000s
I20260806 03:43:51.634817 4626 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.557428 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1442) took 1680 ms. Trace:
I20260806 03:43:51.870518 4640 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.558516 4363 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.558986 4640 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.559340 4363 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.559691 4640 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.517360 564 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:52.037454 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 370) took 2669 ms. Trace:
I20260806 03:43:52.560958 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1299) took 1199 ms. Trace:
W20260806 03:43:51.887392 4554 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.247s user 0.013s sys 0.008s
I20260806 03:43:52.561765 564 rpcz_store.cc:276] 0806 03:43:51.361899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.551733 (+189834us) service_pool.cc:227] Handling call
0806 03:43:52.560874 (+1009141us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:52.358942 4483 consensus_meta.cc:287] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.565s user 0.000s sys 0.021s
I20260806 03:43:52.532569 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:52.043025 4119 raft_consensus.cc:3032] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:51.878123 4577 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.377468 4482 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:52.017253 4576 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.156s user 0.016s sys 0.002s
I20260806 03:43:51.789364 4641 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.532958 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:52.583990 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:52.584177 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:52.584611 564 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:52.286455 4560 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.035s sys 0.031s
W20260806 03:43:52.586723 4560 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.985s user 0.040s sys 0.032s
I20260806 03:43:52.587158 4560 raft_consensus.cc:686] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:52.588748 4626 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.588420 4560 consensus_queue.cc:237] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.589599 4626 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:52.561252 542 rpcz_store.cc:276] 0806 03:43:49.367952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.458571 (+ 90619us) service_pool.cc:227] Handling call
0806 03:43:49.494528 (+ 35957us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:49.551329 (+ 56801us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.037091 (+2485762us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b5400022e50. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:52.037351 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66572544}
I20260806 03:43:52.592530 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:52.592929 4288 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.469s user 0.014s sys 0.004s
I20260806 03:43:52.567296 4119 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:52.594585 4098 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.595140 4098 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.595670 4119 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.557907 769 rpcz_store.cc:276] 0806 03:43:50.877415 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.878728 (+ 1313us) service_pool.cc:227] Handling call
0806 03:43:50.879127 (+ 399us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:50.897943 (+ 18816us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:50.897979 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:50.898580 (+ 601us) log.cc:842] Serialized 26 byte log entry
0806 03:43:50.989193 (+ 90613us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:51.032568 (+ 43375us) log.cc:842] Serialized 14 byte log entry
0806 03:43:51.033101 (+ 533us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.033215 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:52.557177 (+1523962us) raft_consensus.cc:1671] finished
0806 03:43:52.557195 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.557365 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18310,"prepare.queue_time_us":40856,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":19399,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":16355,"thread_start_us":57122,"threads_started":1,"wal-append.queue_time_us":97861}
W20260806 03:43:52.047151 4655 log.cc:924] Time spent T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.391s user 0.002s sys 0.000s
I20260806 03:43:51.650146 4567 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.734957 4351 raft_consensus.cc:2781] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:52.526160 561 rpcz_store.cc:276] 0806 03:43:49.489784 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.530346 (+ 40562us) service_pool.cc:227] Handling call
0806 03:43:49.551546 (+ 21200us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:49.908325 (+356779us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.244711 (+2336386us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400022954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:52.244960 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":729346432}
I20260806 03:43:52.529927 546 rpcz_store.cc:276] 0806 03:43:51.353863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.615554 (+261691us) service_pool.cc:227] Handling call
0806 03:43:52.529315 (+913761us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63232}
I20260806 03:43:51.615029 4620 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:51.735726 4510 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.672894 4563 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.521489 767 rpcz_store.cc:276] 0806 03:43:49.699430 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.793205 (+ 93775us) service_pool.cc:227] Handling call
0806 03:43:50.740754 (+947549us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.741975 (+ 1221us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.119100 (+1377125us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b5400320850. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:52.119331 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2178491264}
I20260806 03:43:52.044162 4423 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:52.570182 671 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:52.448027 4565 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.571358 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:52.538188 4130 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:51.676767 4661 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:52.580252 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.598235 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:52.607434 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36546 (request call id 504) took 1105 ms. Trace:
I20260806 03:43:52.619459 667 rpcz_store.cc:276] 0806 03:43:51.494505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.584968 (+1090463us) service_pool.cc:227] Handling call
0806 03:43:52.599394 (+ 14426us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.620375 769 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 8
W20260806 03:43:52.617324 563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1858) took 1454 ms (client timeout 1667 ms). Trace:
W20260806 03:43:52.621363 563 rpcz_store.cc:269] 0806 03:43:51.163057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.163249 (+ 192us) service_pool.cc:227] Handling call
0806 03:43:52.617116 (+1453867us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b540011e4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:52.617239 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12995456}
I20260806 03:43:52.622078 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 369) took 1177 ms. Trace:
I20260806 03:43:52.622571 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:52.621695 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2310) took 1495 ms. Trace:
I20260806 03:43:52.624523 563 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 10: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:52.566124 4665 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.558106 4650 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.641486 769 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:51.655687 4015 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.599480 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:52.624413 770 rpcz_store.cc:276] 0806 03:43:51.126490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.551564 (+1425074us) service_pool.cc:227] Handling call
0806 03:43:52.621636 (+ 70072us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.622437 671 rpcz_store.cc:276] 0806 03:43:51.445054 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.532438 (+1087384us) service_pool.cc:227] Handling call
0806 03:43:52.569943 (+ 37505us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.621836 (+ 51893us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.622013 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.600978 674 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
W20260806 03:43:52.639117 4498 consensus_meta.cc:287] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.470s user 0.010s sys 0.007s
I20260806 03:43:52.611627 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:52.627148 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:52.625468 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1303) took 1249 ms. Trace:
I20260806 03:43:52.629127 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:52.542729 4578 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.611708 4024 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.667233 561 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:52.667766 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 386) took 1309 ms. Trace:
I20260806 03:43:52.668145 561 rpcz_store.cc:276] 0806 03:43:51.358004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.608762 (+1250758us) service_pool.cc:227] Handling call
0806 03:43:52.667702 (+ 58940us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.668989 4597 consensus_queue.cc:1046] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:52.669361 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:52.670130 4597 raft_consensus.cc:3032] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:52.597536 4152 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.668522 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1313) took 1019 ms. Trace:
I20260806 03:43:52.671408 4152 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:52.593544 4288 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.627332 4423 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.676297 4288 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.618157 4620 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.647712 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1444) took 1758 ms. Trace:
I20260806 03:43:52.650629 667 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260806 03:43:52.677596 4423 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.678298 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1752) took 1174 ms. Trace:
I20260806 03:43:52.678684 667 rpcz_store.cc:276] 0806 03:43:51.504045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.620420 (+1116375us) service_pool.cc:227] Handling call
0806 03:43:52.640904 (+ 20484us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:52.677947 (+ 37043us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.678240 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.615299 4351 raft_consensus.cc:686] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:52.569265 4554 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.630280 4661 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:52.570708 4483 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.616014 4640 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.582159 4482 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.628026 4098 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.690814 563 rpcz_store.cc:276] 0806 03:43:51.376945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.622271 (+1245326us) service_pool.cc:227] Handling call
0806 03:43:52.625401 (+ 3130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.654143 1657 raft_consensus.cc:2454] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:52.670641 4597 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:52.671355 542 rpcz_store.cc:276] 0806 03:43:51.649655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.593949 (+944294us) service_pool.cc:227] Handling call
0806 03:43:52.668417 (+ 74468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.662508 4438 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:52.604586 564 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:52.702827 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:52.618919 4510 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:52.703172 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:52.704247 770 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:52.704303 564 raft_consensus.cc:1261] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:52.677423 773 rpcz_store.cc:276] 0806 03:43:50.889733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.890189 (+ 456us) service_pool.cc:227] Handling call
0806 03:43:50.902879 (+ 12690us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:50.987385 (+ 84506us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:43:50.987418 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:50.988016 (+ 598us) log.cc:842] Serialized 26 byte log entry
0806 03:43:51.054606 (+ 66590us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:51.054657 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.054749 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:52.647415 (+1592666us) raft_consensus.cc:1671] finished
0806 03:43:52.647441 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.647603 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22312,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":11396,"thread_start_us":149795,"threads_started":2,"wal-append.queue_time_us":185654}
I20260806 03:43:52.687701 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2317) took 1555 ms. Trace:
I20260806 03:43:52.705565 767 rpcz_store.cc:276] 0806 03:43:51.132445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.611317 (+1478872us) service_pool.cc:227] Handling call
0806 03:43:52.687639 (+ 76322us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.670159 561 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:52.707487 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1302) took 1342 ms. Trace:
I20260806 03:43:52.707973 561 rpcz_store.cc:276] 0806 03:43:51.364945 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.668999 (+1304054us) service_pool.cc:227] Handling call
0806 03:43:52.707407 (+ 38408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.691985 667 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:52.709322 668 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:52.706703 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.705184 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 388) took 1259 ms. Trace:
I20260806 03:43:52.711258 564 rpcz_store.cc:276] 0806 03:43:51.446176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.703583 (+1257407us) service_pool.cc:227] Handling call
0806 03:43:52.703906 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.704869 (+ 963us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.705118 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.612883 4641 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:52.646488 4255 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.002s sys 0.013s
W20260806 03:43:52.639176 1663 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s user 0.019s sys 0.011s
I20260806 03:43:52.701468 546 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:52.731161 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1874) took 1373 ms. Trace:
I20260806 03:43:52.731701 546 rpcz_store.cc:276] 0806 03:43:51.357398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.680734 (+1323336us) service_pool.cc:227] Handling call
0806 03:43:52.701247 (+ 20513us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.730739 (+ 29492us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.731073 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.590301 4626 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.692691 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:52.710569 767 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:52.768690 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.769330 767 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:43:52.632000 4665 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.775418 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:52.778580 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:52.698629 4268 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:52.706720 4627 consensus_meta.cc:287] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.591s user 0.011s sys 0.006s
I20260806 03:43:52.709865 4438 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:52.790510 616 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:52.780401 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.791882 773 raft_consensus.cc:2379] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:52.792310 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 557) took 1152 ms. Trace:
I20260806 03:43:52.792584 773 rpcz_store.cc:276] 0806 03:43:51.639864 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.705671 (+1065807us) service_pool.cc:227] Handling call
0806 03:43:52.792253 (+ 86582us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.792866 667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 503) took 1677 ms. Trace:
I20260806 03:43:52.793793 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:52.786772 4670 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260806 03:43:52.795769 4578 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.786506 4597 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.797396 4597 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:52.797684 4015 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.798259 4554 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.687856 4498 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:52.729928 4255 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.715s user 0.002s sys 0.013s
W20260806 03:43:52.766816 1663 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.882s user 0.022s sys 0.011s
I20260806 03:43:52.787153 728 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:52.789134 546 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:52.783741 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1305) took 1220 ms. Trace:
I20260806 03:43:52.802052 616 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:52.796160 1625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.409s user 0.023s sys 0.023s
I20260806 03:43:52.782590 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2308) took 1657 ms. Trace:
I20260806 03:43:52.804785 4626 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:52.691355 4661 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.805621 1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 448) took 1560 ms. Trace:
I20260806 03:43:52.806829 4482 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.807494 4665 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.787734 4268 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:52.801278 4438 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:52.787822 4627 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.803032 773 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:52.810745 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1451) took 1715 ms. Trace:
I20260806 03:43:52.814050 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1438) took 2031 ms. Trace:
I20260806 03:43:52.793403 667 rpcz_store.cc:276] 0806 03:43:51.115418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.776840 (+1661422us) service_pool.cc:227] Handling call
0806 03:43:52.792798 (+ 15958us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:52.732568 4510 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:52.708853 4351 consensus_queue.cc:237] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.009357 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 382) took 3268 ms. Trace:
I20260806 03:43:52.834249 770 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:53.058961 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1455) took 1503 ms. Trace:
I20260806 03:43:53.041656 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2307) took 1916 ms. Trace:
W20260806 03:43:53.006986 769 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2319) took 1873 ms (client timeout 2242 ms). Trace:
W20260806 03:43:53.002262 674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1737) took 2820 ms (client timeout 1667 ms). Trace:
I20260806 03:43:53.059372 770 rpcz_store.cc:276] 0806 03:43:51.555539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.683921 (+1128382us) service_pool.cc:227] Handling call
0806 03:43:53.058902 (+374981us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:53.060073 1625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.772s user 0.029s sys 0.034s
I20260806 03:43:53.060351 776 rpcz_store.cc:276] 0806 03:43:51.125240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.127870 (+ 2630us) service_pool.cc:227] Handling call
0806 03:43:51.568020 (+440150us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:51.568650 (+ 630us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.041366 (+1472716us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54001022d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.041573 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1011331968}
W20260806 03:43:53.046627 1670 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.011s sys 0.000s
W20260806 03:43:53.061726 1670 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.530s user 0.012s sys 0.000s
I20260806 03:43:53.062175 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2314) took 1931 ms. Trace:
I20260806 03:43:53.059118 767 rpcz_store.cc:276] 0806 03:43:51.125697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.781525 (+1655828us) service_pool.cc:227] Handling call
0806 03:43:52.781919 (+ 394us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.782168 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:52.782204 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:52.782258 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:52.782308 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.782387 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.782510 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.062762 776 rpcz_store.cc:276] 0806 03:43:51.131041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.061430 (+1930389us) service_pool.cc:227] Handling call
0806 03:43:53.061686 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.061866 (+ 180us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.061892 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.061934 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:53.061965 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.062013 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.062123 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.064008 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1452) took 1919 ms. Trace:
W20260806 03:43:52.990504 729 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.009s sys 0.034s
W20260806 03:43:53.064719 729 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s user 0.014s sys 0.035s
W20260806 03:43:53.059626 769 rpcz_store.cc:269] 0806 03:43:51.133224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.598008 (+1464784us) service_pool.cc:227] Handling call
0806 03:43:53.006759 (+408751us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b5400312754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.006907 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13047680}
I20260806 03:43:53.066776 767 rpcz_store.cc:276] 0806 03:43:51.144631 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.063281 (+1918650us) service_pool.cc:227] Handling call
0806 03:43:53.063549 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.063707 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.063728 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.063771 (+ 43us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:53.063802 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.063847 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.063945 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:53.068948 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.006s sys 0.002s
W20260806 03:43:53.069406 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s user 0.006s sys 0.002s
W20260806 03:43:53.060143 674 rpcz_store.cc:269] 0806 03:43:50.181725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.208001 (+ 26276us) service_pool.cc:227] Handling call
0806 03:43:53.002012 (+2794011us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540010f9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.002172 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5308265856}
I20260806 03:43:53.059310 772 rpcz_store.cc:276] 0806 03:43:50.783100 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.783318 (+ 218us) service_pool.cc:227] Handling call
0806 03:43:50.807427 (+ 24109us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:51.607878 (+800451us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.813729 (+1205851us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b5400321754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:52.813975 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1791886080}
W20260806 03:43:52.887041 4473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.015s sys 0.023s
I20260806 03:43:53.095991 776 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:53.096762 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 554) took 1863 ms. Trace:
I20260806 03:43:53.097152 776 rpcz_store.cc:276] 0806 03:43:51.233258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.095472 (+1862214us) service_pool.cc:227] Handling call
0806 03:43:53.095826 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.096549 (+ 723us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.096714 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19712}
I20260806 03:43:53.098202 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.099268 776 raft_consensus.cc:2396] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:52.818229 4363 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.096350 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1453) took 1746 ms. Trace:
I20260806 03:43:53.032533 4641 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:52.994820 4636 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 03:43:52.999487 4671 log.cc:924] Time spent T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260806 03:43:53.000968 4498 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:52.888720 4015 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:43:53.057363 1660 rpcz_store.cc:276] 0806 03:43:51.244997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.269816 (+ 24819us) service_pool.cc:227] Handling call
0806 03:43:51.326749 (+ 56933us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:51.397722 (+ 70973us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:51.397807 (+ 85us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:51.554171 (+156364us) log.cc:842] Serialized 52 byte log entry
0806 03:43:51.836432 (+282261us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:51.839671 (+ 3239us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.839779 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:52.805348 (+965569us) raft_consensus.cc:1671] finished
0806 03:43:52.805378 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.805545 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":123056,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":264,"thread_start_us":229449,"threads_started":1,"wal-append.queue_time_us":1107182}
W20260806 03:43:53.131276 4515 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260806 03:43:53.124433 4640 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:53.079736 4661 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:53.082547 4644 log.cc:924] Time spent T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 2.220s user 0.008s sys 0.007s
I20260806 03:43:53.083562 4689 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:53.132006 4640 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.132970 4689 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.134538 4687 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:53.135035 4689 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:53.057519 775 rpcz_store.cc:276] 0806 03:43:51.095845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.096005 (+ 160us) service_pool.cc:227] Handling call
0806 03:43:51.122171 (+ 26166us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:51.125034 (+ 2863us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:51.125089 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:51.125861 (+ 772us) log.cc:842] Serialized 26 byte log entry
0806 03:43:52.110712 (+984851us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b18003558a0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa6411 00007f32162f35cf
0806 03:43:52.566928 (+456216us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:52.566968 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.567030 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:52.802468 (+235438us) raft_consensus.cc:1671] finished
0806 03:43:52.802491 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.810673 (+ 8182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2184,"raft.queue_time_us":7108,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":22353,"spinlock_wait_cycles":22265600,"thread_start_us":456013,"threads_started":1,"wal-append.queue_time_us":1643686}
I20260806 03:43:53.085601 3902 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:43:53.136263 3902 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:53.136679 3902 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:53.135061 4687 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.083990 4626 raft_consensus.cc:2781] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:52.810746 4567 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.138777 4626 raft_consensus.cc:686] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:52.818979 4483 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.139765 4427 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:53.139729 4626 consensus_queue.cc:237] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.140467 4427 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.087765 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.142752 773 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:53.134256 4640 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.138700 4687 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.144407 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 556) took 1544 ms. Trace:
I20260806 03:43:53.144786 773 rpcz_store.cc:276] 0806 03:43:51.600151 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.143581 (+1543430us) service_pool.cc:227] Handling call
0806 03:43:53.143965 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.144143 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.144169 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.144203 (+ 34us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:53.144230 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.144274 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.144362 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.094659 4417 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:52.846441 4569 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.827s user 0.018s sys 0.002s
I20260806 03:43:53.039913 4566 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:53.099380 3906 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s user 0.011s sys 0.000s
I20260806 03:43:53.183466 4510 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:53.001333 4591 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:53.185837 4152 consensus_meta.cc:287] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.514s user 0.005s sys 0.004s
I20260806 03:43:52.951805 4562 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:53.111464 3905 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.005s sys 0.006s
W20260806 03:43:53.095391 4473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.027s sys 0.047s
I20260806 03:43:53.169322 4417 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.135680 4689 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:53.171574 4427 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.118796 4636 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:53.058223 544 rpcz_store.cc:276] 0806 03:43:49.741303 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.741440 (+ 137us) service_pool.cc:227] Handling call
0806 03:43:49.741762 (+ 322us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:49.764103 (+ 22341us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:49.764132 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:49.764555 (+ 423us) log.cc:842] Serialized 26 byte log entry
0806 03:43:49.896798 (+132243us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:49.969881 (+ 73083us) log.cc:842] Serialized 14 byte log entry
0806 03:43:49.972297 (+ 2416us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:49.986440 (+ 14143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:51.079964 (+1093524us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b180035a100. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f321336e4ca 00007f321ca4ff99 00007f321a1c7bcf 00007f32133f77b6 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d
0806 03:43:53.003003 (+1923039us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b180035a100. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:53.009061 (+ 6058us) raft_consensus.cc:1671] finished
0806 03:43:53.009090 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.009281 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24400,"prepare.queue_time_us":125879,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":22737,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":45119,"spinlock_wait_cycles":7857280,"thread_start_us":122279,"threads_started":2,"wal-append.queue_time_us":189698}
I20260806 03:43:53.077649 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1306) took 1502 ms. Trace:
I20260806 03:43:53.087917 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:53.076561 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:53.061185 564 rpcz_store.cc:276] 0806 03:43:51.567489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.712586 (+1145097us) service_pool.cc:227] Handling call
0806 03:43:52.783260 (+ 70674us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.783439 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:52.783461 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:52.783502 (+ 41us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:52.783539 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:52.783592 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:52.783688 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:52.888609 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 45642.3us
W20260806 03:43:53.085978 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 284013us
I20260806 03:43:53.084146 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2359) took 3508 ms. Trace:
I20260806 03:43:53.086318 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1429) took 2937 ms. Trace:
W20260806 03:43:53.080374 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 293304us
I20260806 03:43:53.068451 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1454) took 1687 ms. Trace:
I20260806 03:43:53.111491 769 rpcz_store.cc:276] 0806 03:43:51.350270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.095479 (+1745209us) service_pool.cc:227] Handling call
0806 03:43:53.095881 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.096050 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.096076 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.096112 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:53.096147 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.096197 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.096300 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188672}
I20260806 03:43:53.090641 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2301) took 3345 ms. Trace:
I20260806 03:43:53.125329 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1456) took 1551 ms. Trace:
I20260806 03:43:53.146281 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 558) took 1347 ms. Trace:
I20260806 03:43:52.516053 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:43:53.508769 614 connection.cc:232] RPC call timeout handler was delayed by -1.2891492950730026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260806 03:43:53.563369 614 connection.cc:232] RPC call timeout handler was delayed by -1.181149295065552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260806 03:43:53.570202 614 connection.cc:232] RPC call timeout handler was delayed by -1.0920492950826883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260806 03:43:53.480327 730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.008s sys 0.008s
W20260806 03:43:53.446897 547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1864) took 2272 ms (client timeout 1613 ms). Trace:
W20260806 03:43:53.476639 671 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.006s sys 0.002s
W20260806 03:43:53.416519 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.871s user 0.009s sys 0.004s
W20260806 03:43:53.520447 1622 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.007s sys 0.009s
I20260806 03:43:53.418682 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1871) took 2117 ms. Trace:
W20260806 03:43:53.573073 614 connection.cc:232] RPC call timeout handler was delayed by -1.0812492950353771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
I20260806 03:43:53.516216 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2311) took 2387 ms. Trace:
W20260806 03:43:53.574244 614 connection.cc:232] RPC call timeout handler was delayed by -1.072249294957146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20550000000000002s for a response
W20260806 03:43:53.574770 614 connection.cc:232] RPC call timeout handler was delayed by -1.0146492950152606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
W20260806 03:43:53.478281 668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2385) took 2365 ms (client timeout 1757 ms). Trace:
W20260806 03:43:53.478279 665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2370) took 3781 ms (client timeout 3205 ms). Trace:
I20260806 03:43:53.479405 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 357) took 3771 ms. Trace:
W20260806 03:43:53.425174 561 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.004s sys 0.000s
W20260806 03:43:53.416584 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.871s user 0.007s sys 0.004s
I20260806 03:43:53.447685 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 358) took 3722 ms. Trace:
W20260806 03:43:53.416803 563 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.005s sys 0.027s
W20260806 03:43:53.417071 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.848s user 0.029s sys 0.037s
I20260806 03:43:53.587955 4685 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:53.586756 766 rpcz_store.cc:276] 0806 03:43:50.149130 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.247108 (+ 97978us) service_pool.cc:227] Handling call
0806 03:43:50.247359 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.247559 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:43:50.282183 (+ 34624us) log.cc:842] Serialized 14 byte log entry
0806 03:43:50.419532 (+137349us) log.cc:842] Serialized 14 byte log entry
0806 03:43:53.085901 (+2666369us) spinlock_profiling.cc:243] Waited 6.38 ms on lock 0x7b180047b4e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f3215fae82c 00007f32133d99dc 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216329576 00007f321633a0d7
0806 03:43:53.085995 (+ 94us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.086046 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:53.086097 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.086163 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.086263 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68460,"prepare.queue_time_us":54938,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":124,"raft.queue_time_us":433326,"raft.run_cpu_time_us":446,"raft.run_wall_time_us":24849,"spinlock_wait_cycles":14684416,"thread_start_us":256250,"threads_started":3}
I20260806 03:43:53.590595 4114 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:53.592015 4114 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.487411 664 raft_consensus.cc:1226] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:43:53.146474 4015 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:53.451097 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:53.495798 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.595305 4015 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:53.595441 376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 144679us
I20260806 03:43:53.596378 772 raft_consensus.cc:2379] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:53.602744 1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 836) took 3767 ms. Trace:
I20260806 03:43:53.603645 546 rpcz_store.cc:276] 0806 03:43:51.575603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.732742 (+1157139us) service_pool.cc:227] Handling call
0806 03:43:52.788937 (+ 56195us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.077307 (+288370us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.077577 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.605453 4119 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:53.573951 770 rpcz_store.cc:276] 0806 03:43:51.126846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.060351 (+1933505us) service_pool.cc:227] Handling call
0806 03:43:53.060682 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.120348 (+ 59666us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.120392 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.120460 (+ 68us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:53.120496 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.513786 (+393290us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b5400055ad4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.513872 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.514223 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136251776}
W20260806 03:43:53.583813 1622 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.676s user 0.009s sys 0.009s
I20260806 03:43:53.612000 766 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 6. Ops: []
W20260806 03:43:53.619338 730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.101s user 0.010s sys 0.013s
I20260806 03:43:53.586990 773 rpcz_store.cc:276] 0806 03:43:51.799423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.145587 (+1346164us) service_pool.cc:227] Handling call
0806 03:43:53.145846 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.146017 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.146038 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.146071 (+ 33us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:53.146101 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.146143 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.146234 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.479295 663 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:53.585359 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.040s user 0.016s sys 0.010s
W20260806 03:43:53.625204 671 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.847s user 0.007s sys 0.011s
I20260806 03:43:53.577948 669 rpcz_store.cc:276] 0806 03:43:49.707483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.047332 (+339849us) service_pool.cc:227] Handling call
0806 03:43:50.047692 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.106982 (+ 59290us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.478803 (+3371821us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b540016dfd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.479217 (+ 414us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":118299008}
I20260806 03:43:53.602504 767 rpcz_store.cc:276] 0806 03:43:51.381204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.067761 (+1686557us) service_pool.cc:227] Handling call
0806 03:43:53.068030 (+ 269us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.068193 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.068215 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.068249 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:53.068279 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.068323 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.068407 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.618136 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1308) took 2026 ms. Trace:
W20260806 03:43:53.631577 561 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s user 0.010s sys 0.002s
I20260806 03:43:53.623328 772 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5. Ops: []
W20260806 03:43:53.576421 665 rpcz_store.cc:269] 0806 03:43:49.696523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.961172 (+264649us) service_pool.cc:227] Handling call
0806 03:43:53.477758 (+3516586us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b5400386b54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.478018 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2048190848}
I20260806 03:43:53.651482 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 559) took 1650 ms. Trace:
I20260806 03:43:53.562093 661 rpcz_store.cc:276] 0806 03:43:49.576015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.578556 (+ 2541us) service_pool.cc:227] Handling call
0806 03:43:49.578794 (+ 238us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.871160 (+1292366us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.083876 (+2212716us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540015b8d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.084083 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2950839680}
W20260806 03:43:53.576047 668 rpcz_store.cc:269] 0806 03:43:51.112395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.117955 (+ 5560us) service_pool.cc:227] Handling call
0806 03:43:53.477658 (+2359703us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540015ccd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.478018 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3490621184}
W20260806 03:43:53.631881 563 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s user 0.009s sys 0.035s
W20260806 03:43:53.571743 547 rpcz_store.cc:269] 0806 03:43:51.174000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.174600 (+ 600us) service_pool.cc:227] Handling call
0806 03:43:53.446171 (+2271571us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540011dfd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.446530 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4930939136}
W20260806 03:43:53.627117 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.058s user 0.038s sys 0.041s
W20260806 03:43:53.614645 1670 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s user 0.001s sys 0.009s
I20260806 03:43:52.940258 542 raft_consensus.cc:1261] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 03:43:53.479321 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:53.579173 659 rpcz_store.cc:276] 0806 03:43:49.725339 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.049216 (+323877us) service_pool.cc:227] Handling call
0806 03:43:50.067744 (+ 18528us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.097543 (+ 29799us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.447090 (+3349547us) spinlock_profiling.cc:243] Waited 959 us on lock 0x7b540019b9d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.447372 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2206464}
I20260806 03:43:53.668732 766 rpcz_store.cc:276] 0806 03:43:51.982970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.590350 (+1607380us) service_pool.cc:227] Handling call
0806 03:43:53.611559 (+ 21209us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.632622 (+ 21063us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.632846 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:53.670939 401 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35388 (request call id 319) took 19632 ms (client timeout 15000 ms). Trace:
I20260806 03:43:53.663122 4528 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:53.671542 3905 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.086s user 0.013s sys 0.015s
I20260806 03:43:53.189518 4569 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.658293 4268 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.663789 4547 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:53.668051 564 rpcz_store.cc:276] 0806 03:43:51.591996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.616266 (+2024270us) service_pool.cc:227] Handling call
0806 03:43:53.617012 (+ 746us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.617428 (+ 416us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.617458 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.617535 (+ 77us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:53.617609 (+ 74us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.617704 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.617865 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.628073 4431 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:53.683729 776 rpcz_store.cc:276] 0806 03:43:51.573736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.101226 (+1527490us) service_pool.cc:227] Handling call
0806 03:43:53.124702 (+ 23476us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.124914 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.124946 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.125010 (+ 64us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:53.125054 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.125112 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.125259 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.679775 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 560) took 1576 ms. Trace:
I20260806 03:43:53.289386 4558 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:53.212312 4562 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.690578 4558 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:53.671574 771 rpcz_store.cc:276] 0806 03:43:49.745193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:49.817159 (+ 71966us) service_pool.cc:227] Handling call
0806 03:43:49.817563 (+ 404us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:49.913846 (+ 96283us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:49.913874 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:49.914459 (+ 585us) log.cc:842] Serialized 26 byte log entry
0806 03:43:49.966213 (+ 51754us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:49.966248 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.090154 (+3123906us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b5400322154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.090198 (+ 44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:53.090246 (+ 48us) raft_consensus.cc:1671] finished
0806 03:43:53.090257 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.090550 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6664,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":160,"spinlock_wait_cycles":220612736,"thread_start_us":5659,"threads_started":1,"wal-append.queue_time_us":44354,"wal-append.run_cpu_time_us":4681,"wal-append.run_wall_time_us":1296471}
I20260806 03:43:53.692512 771 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:53.692766 4562 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:53.692265 4558 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.589421 4685 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.688921 772 rpcz_store.cc:276] 0806 03:43:52.092482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.597343 (+1504861us) service_pool.cc:227] Handling call
0806 03:43:53.598043 (+ 700us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.668344 (+ 70301us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.668544 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:53.664935 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 757662us
I20260806 03:43:53.286656 4627 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:53.704022 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 1.13003e+06us
I20260806 03:43:53.721179 548 raft_consensus.cc:2396] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:53.721735 4483 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:53.721025 4626 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.045s sys 0.054s
W20260806 03:43:53.722590 4626 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s user 0.046s sys 0.054s
I20260806 03:43:53.722899 4483 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.724145 4417 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.724545 614 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:53.726866 4688 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:53.726760 4528 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.727669 4688 raft_consensus.cc:686] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:53.729729 4528 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.670846 4098 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:53.730882 4268 consensus_queue.cc:1035] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:53.731047 4098 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.732542 4427 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 03:43:53.736415 1670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 846) took 1216 ms. Trace:
I20260806 03:43:53.733819 4268 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:43:53.726877 4431 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.761193 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.729310 4688 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.603500 1663 rpcz_store.cc:276] 0806 03:43:49.834602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.194617 (+360015us) service_pool.cc:227] Handling call
0806 03:43:50.252992 (+ 58375us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:50.306723 (+ 53731us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:50.306762 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:51.883936 (+1577174us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b2c001519d8. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:43:51.884488 (+ 552us) log.cc:842] Serialized 26 byte log entry
0806 03:43:53.111582 (+1227094us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:53.120153 (+ 8571us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.130868 (+ 10715us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:53.147106 (+ 16238us) raft_consensus.cc:1671] finished
0806 03:43:53.160156 (+ 13050us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.583214 (+423058us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16065,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":157,"spinlock_wait_cycles":5043968,"thread_start_us":1172432,"threads_started":1,"wal-append.queue_time_us":1064171}
I20260806 03:43:53.694864 4547 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:53.722123 548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1315) took 1934 ms (client timeout 1565 ms). Trace:
I20260806 03:43:53.664383 4636 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.664654 4689 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.749576 4558 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:53.754287 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:53.767488 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1309) took 2166 ms. Trace:
I20260806 03:43:53.749310 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:53.754755 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:53.756927 768 raft_consensus.cc:2354] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:53.725914 4483 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.739192 4685 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:53.755101 546 pending_rounds.cc:76] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 2
I20260806 03:43:53.812381 546 pending_rounds.cc:97] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 03:43:53.812819 546 raft_consensus.cc:2866] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:43:53.815269 775 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.039s sys 0.043s
W20260806 03:43:53.815830 775 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s user 0.039s sys 0.043s
I20260806 03:43:53.816129 4547 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.572917 562 rpcz_store.cc:276] 0806 03:43:51.301446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.312970 (+ 11524us) service_pool.cc:227] Handling call
0806 03:43:51.342950 (+ 29980us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:51.363177 (+ 20227us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:51.363224 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:51.363873 (+ 649us) log.cc:842] Serialized 52 byte log entry
0806 03:43:51.650280 (+286407us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:51.650442 (+ 162us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:51.650601 (+ 159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:53.417639 (+1767038us) raft_consensus.cc:1671] finished
0806 03:43:53.417740 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.418299 (+ 559us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30811,"raft.queue_time_us":19869,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":31628,"thread_start_us":95372,"threads_started":1,"wal-append.queue_time_us":1391991}
W20260806 03:43:53.817144 4566 consensus_meta.cc:287] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.628s user 0.013s sys 0.004s
I20260806 03:43:53.743860 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:53.756987 1657 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:53.762853 542 pending_rounds.cc:76] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 1
W20260806 03:43:53.757095 558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1294) took 2314 ms (client timeout 2161 ms). Trace:
I20260806 03:43:53.762142 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:53.759333 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 439) took 3709 ms. Trace:
I20260806 03:43:53.763337 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:53.819515 669 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:53.763893 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1457) took 1069 ms. Trace:
I20260806 03:43:53.781674 4626 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:53.766079 663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1750) took 2674 ms. Trace:
W20260806 03:43:53.675892 401 rpcz_store.cc:269] 0806 03:43:34.081180 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.140820 (+ 59640us) service_pool.cc:227] Handling call
0806 03:43:53.666415 (+19525595us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:46.330435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:46.331132 (+ 697us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:46.331185 (+ 53us) write_op.cc:435] Acquired schema lock
0806 03:43:46.331202 (+ 17us) tablet.cc:660] Decoding operations
0806 03:43:46.390037 (+ 58835us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:46.449483 (+ 59446us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:46.449524 (+ 41us) tablet.cc:683] Acquiring locks for 17 operations
0806 03:43:46.451036 (+ 1512us) tablet.cc:699] Row locks acquired
0806 03:43:46.451073 (+ 37us) write_op.cc:260] PREPARE: finished
0806 03:43:46.451255 (+ 182us) write_op.cc:270] Start()
0806 03:43:46.451417 (+ 162us) write_op.cc:276] Timestamp: P: 1785987826451179 usec, L: 0
0806 03:43:46.451454 (+ 37us) op_driver.cc:331] REPLICATION: starting
0806 03:43:46.459865 (+ 8411us) log.cc:842] Serialized 6721 byte log entry
0806 03:43:47.307605 (+847740us) op_driver.cc:444] REPLICATION: finished
0806 03:43:47.835358 (+527753us) write_op.cc:301] APPLY: starting
0806 03:43:47.835465 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:47.835875 (+ 410us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:47.835993 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:47.838323 (+ 2330us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:47.881868 (+ 43545us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0806 03:43:47.881951 (+ 83us) write_op.cc:312] APPLY: finished
0806 03:43:47.882598 (+ 647us) log.cc:842] Serialized 119 byte log entry
0806 03:43:47.884417 (+ 1819us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:47.884660 (+ 243us) write_op.cc:454] Released schema lock
0806 03:43:47.884758 (+ 98us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:47.885134 (+ 376us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:47.885164 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":382239,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":434124,"apply.run_cpu_time_us":5923,"apply.run_wall_time_us":50268,"num_ops":17,"prepare.queue_time_us":349170,"prepare.run_cpu_time_us":43600,"prepare.run_wall_time_us":244750,"raft.queue_time_us":275930,"raft.run_cpu_time_us":154288,"raft.run_wall_time_us":733000,"replication_time_us":783946,"thread_start_us":1004937,"threads_started":4,"wal-append.queue_time_us":147722,"wal-append.run_cpu_time_us":14882,"wal-append.run_wall_time_us":2249731}]]}
I20260806 03:43:53.782509 772 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
W20260806 03:43:53.804728 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 92398.7us
I20260806 03:43:53.470587 403 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 5, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:53.807751 766 raft_consensus.cc:1226] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 2. Ops: []
I20260806 03:43:53.800954 4562 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:53.809839 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 331171us
W20260806 03:43:53.809952 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 103698us
I20260806 03:43:53.810220 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.801080 544 rpcz_store.cc:276] 0806 03:43:51.600906 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.682976 (+2082070us) service_pool.cc:227] Handling call
0806 03:43:53.766672 (+ 83696us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.767044 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.767071 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.767116 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:53.767147 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.767200 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.767310 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:43:53.796193 548 rpcz_store.cc:269] 0806 03:43:51.788182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.512224 (+724042us) service_pool.cc:227] Handling call
0806 03:43:53.087776 (+575552us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5800003398. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:53.721999 (+634223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30085504}
I20260806 03:43:53.736923 1670 rpcz_store.cc:276] 0806 03:43:52.520081 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.520249 (+ 168us) service_pool.cc:227] Handling call
0806 03:43:52.520599 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.520841 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:43:53.062122 (+541281us) log.cc:842] Serialized 14 byte log entry
0806 03:43:53.290693 (+228571us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b180041a820. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:53.735431 (+444738us) log.cc:842] Serialized 14 byte log entry
0806 03:43:53.735783 (+ 352us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.735850 (+ 67us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:53.735948 (+ 98us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.736075 (+ 127us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.736305 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":751328,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":1622,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":192,"spinlock_wait_cycles":4290816,"thread_start_us":1149750,"threads_started":2,"wal-append.queue_time_us":221586}
I20260806 03:43:53.828191 4575 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:53.824247 1662 rpcz_store.cc:276] 0806 03:43:50.049500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:50.246615 (+197115us) service_pool.cc:227] Handling call
0806 03:43:50.246937 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:50.787257 (+540320us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.679845 (+2892588us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400696ed4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.759112 (+ 79267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":977963264}
I20260806 03:43:53.829288 767 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:53.832258 4015 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:53.819964 769 rpcz_store.cc:276] 0806 03:43:52.694710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.605026 (+910316us) service_pool.cc:227] Handling call
0806 03:43:53.605643 (+ 617us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.738100 (+132457us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.738133 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:53.738171 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:53.738199 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.763448 (+ 25249us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54002bd9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:53.763518 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.763681 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":303928064}
I20260806 03:43:53.836762 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:53.818761 558 rpcz_store.cc:269] 0806 03:43:51.352745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.620477 (+267732us) service_pool.cc:227] Handling call
0806 03:43:53.076419 (+1455942us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b5800003398. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bfbce 000000000034aea9 00007f321b1eba30 00007f321b2b17b4 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:43:53.658819 (+582400us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10324736}
I20260806 03:43:53.775544 4268 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:53.734163 4098 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:53.825152 663 rpcz_store.cc:276] 0806 03:43:51.092002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.101859 (+ 9857us) service_pool.cc:227] Handling call
0806 03:43:53.478652 (+2376793us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.680707 (+202055us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.765842 (+ 85135us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400386b50. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:53.766012 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5406716160}
I20260806 03:43:53.859467 1664 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:53.756217 4569 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:53.853139 674 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:53.828183 542 pending_rounds.cc:97] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:43:53.875754 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 555) took 2276 ms. Trace:
I20260806 03:43:53.878615 4562 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.862880 4566 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:53.868247 403 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:53.883679 4547 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:53.890710 669 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:53.884384 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:53.885106 4510 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:53.886874 775 rpcz_store.cc:276] 0806 03:43:51.599393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.136335 (+1536942us) service_pool.cc:227] Handling call
0806 03:43:53.136710 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:53.137050 (+ 340us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.137079 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:53.137517 (+ 438us) log.cc:842] Serialized 26 byte log entry
0806 03:43:53.816443 (+678926us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:53.816520 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.816634 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:53.875467 (+ 58833us) raft_consensus.cc:1671] finished
0806 03:43:53.875498 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:53.875662 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1079,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":678340,"threads_started":1,"wal-append.queue_time_us":730146}
I20260806 03:43:53.855913 4626 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:53.915079 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.370s user 0.008s sys 0.017s
I20260806 03:43:53.868010 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:53.890895 542 raft_consensus.cc:2866] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:53.933107 663 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:53.932278 547 raft_consensus.cc:1226] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:53.942971 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:53.927732 4563 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.022s sys 0.085s
W20260806 03:43:53.940591 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.39043e+06us
I20260806 03:43:53.958425 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1316) took 1436 ms. Trace:
I20260806 03:43:53.958963 547 rpcz_store.cc:276] 0806 03:43:52.508585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.818206 (+1309621us) service_pool.cc:227] Handling call
0806 03:43:53.897299 (+ 79093us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.940949 (+ 43650us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:53.944974 (+ 4025us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:53.887029 4575 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:53.932229 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:53.939672 4636 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:53.975201 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 231570us
W20260806 03:43:53.966753 4563 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.023s sys 0.085s
I20260806 03:43:53.968736 3905 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:53.982630 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:53.988834 4376 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:53.989570 4658 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:53.988745 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 239884us
I20260806 03:43:53.987680 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:53.945943 4566 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:54.011628 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 142803us
I20260806 03:43:54.012748 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.014830 4405 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:54.016057 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 67064.3us
I20260806 03:43:54.016776 4015 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.009629 403 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.018664 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1304) took 2467 ms. Trace:
I20260806 03:43:54.018606 3905 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.019323 542 rpcz_store.cc:276] 0806 03:43:51.551696 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.729618 (+1177922us) service_pool.cc:227] Handling call
0806 03:43:52.782836 (+ 53218us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:53.985097 (+1202261us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.018587 (+ 33490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:54.019021 403 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:54.027678 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.028373 766 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:54.028004 4626 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.030812 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:54.031826 498 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:54.032250 4427 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:43:54.032830 1622 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.033625 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:54.033507 4427 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 03:43:54.035621 4427 raft_consensus.cc:3032] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:54.036022 4427 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:54.036235 4268 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.037114 4427 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.037871 4268 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:54.038677 4427 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.039702 4431 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.041178 4689 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.042104 4473 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:54.042681 4473 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:54.042874 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1751) took 2870 ms. Trace:
I20260806 03:43:54.043103 4473 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:54.043292 664 rpcz_store.cc:276] 0806 03:43:51.172962 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:51.623301 (+450339us) service_pool.cc:227] Handling call
0806 03:43:53.486673 (+1863372us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:53.594767 (+108094us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.042648 (+447881us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b540015ccd0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:54.042811 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4110694784}
I20260806 03:43:53.990332 4658 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.049244 4658 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.049420 4098 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:54.050017 661 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:54.051033 614 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:54.050117 4098 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:54.052238 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 56639.8us
I20260806 03:43:54.053166 614 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:54.032949 772 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:54.060808 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.059710 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.018249 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:54.054703 1664 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:54.054981 4119 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:43:54.020828 403 catalog_manager.cc:5495] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.071596 1660 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:54.083453 498 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.060884 4427 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.083791 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.084750 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.085812 4427 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.087136 1670 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 3.
I20260806 03:43:54.087245 4405 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:43:54.088052 4405 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:43:54.088593 4405 raft_consensus.cc:686] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:54.089344 4427 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.054287 4098 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.061101 4422 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:54.090889 4497 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260806 03:43:54.091300 768 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 553) took 2920 ms (client timeout 1963 ms). Trace:
I20260806 03:43:54.091501 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:54.092262 767 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:54.004315 4376 raft_consensus.cc:686] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:54.090935 4422 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:54.093134 4422 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:54.093969 4376 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.095585 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.060633 4562 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:54.097000 4438 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.025s user 0.014s sys 0.008s
I20260806 03:43:54.097474 4438 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.099665 4438 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.084926 665 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:54.100894 4473 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.102136 4114 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.096549 775 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:54.102802 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:54.103367 4602 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:54.054759 4431 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:54.104773 3905 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.056746 4566 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:54.106310 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 88200.8us
W20260806 03:43:54.091856 768 rpcz_store.cc:269] 0806 03:43:51.170900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.517052 (+1346152us) service_pool.cc:227] Handling call
0806 03:43:54.090977 (+1573925us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540030ff54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:54.091188 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2790825856}
W20260806 03:43:54.108934 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 96408.2us
I20260806 03:43:54.108295 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60246 (request call id 625) took 1416 ms. Trace:
I20260806 03:43:53.978535 4510 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.111127 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.113154 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:54.116510 1657 rpcz_store.cc:276] 0806 03:43:52.692273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.692379 (+ 106us) service_pool.cc:227] Handling call
0806 03:43:54.108047 (+1415668us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400525654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:54.108217 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1901057280}
I20260806 03:43:54.110646 4427 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:54.110469 1663 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:54.089722 4405 consensus_queue.cc:237] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.123970 403 catalog_manager.cc:5495] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 6, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 6 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.108593 4422 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.125245 4427 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:54.138425 4431 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:54.140652 4602 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:54.150141 4427 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:54.143071 4558 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.151690 4114 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.176293 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 389) took 2619 ms. Trace:
I20260806 03:43:54.177181 4575 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.158831 4422 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.163637 4663 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.188520 1624 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.179739 561 rpcz_store.cc:276] 0806 03:43:51.557371 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.708907 (+1151536us) service_pool.cc:227] Handling call
0806 03:43:52.783165 (+ 74258us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:52.786027 (+ 2862us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:52.786055 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:52.794743 (+ 8688us) log.cc:842] Serialized 52 byte log entry
0806 03:43:53.808246 (+1013503us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:54.139566 (+331320us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.175834 (+ 36268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.175968 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:54.176047 (+ 79us) raft_consensus.cc:1671] finished
0806 03:43:54.176057 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:54.176231 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34692,"raft.queue_time_us":236683,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":180,"thread_start_us":1205540,"threads_started":2,"wal-append.queue_time_us":1081586}
W20260806 03:43:54.204049 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 125854us
I20260806 03:43:54.197934 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1310) took 2589 ms. Trace:
I20260806 03:43:54.221324 4558 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.222942 4473 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.222592 4663 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.225099 4602 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.226759 403 catalog_manager.cc:5495] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.228283 403 catalog_manager.cc:5495] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:54.212519 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1301) took 2847 ms. Trace:
I20260806 03:43:54.206022 564 rpcz_store.cc:276] 0806 03:43:51.603222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.708645 (+2105423us) service_pool.cc:227] Handling call
0806 03:43:53.845511 (+136866us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:53.880032 (+ 34521us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.880068 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:53.880777 (+ 709us) log.cc:842] Serialized 52 byte log entry
0806 03:43:54.170051 (+289274us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:43:54.170445 (+ 394us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.170682 (+ 237us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:54.191960 (+ 21278us) raft_consensus.cc:1671] finished
0806 03:43:54.191986 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:54.192188 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":461,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":288779,"threads_started":1,"wal-append.queue_time_us":295091}
W20260806 03:43:54.234495 3902 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.031s user 0.013s sys 0.005s
I20260806 03:43:54.230346 403 catalog_manager.cc:5495] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:43:54.236379 563 rpcz_store.cc:276] 0806 03:43:51.364658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.691591 (+1326933us) service_pool.cc:227] Handling call
0806 03:43:52.765214 (+ 73623us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:52.779208 (+ 13994us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:52.779249 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:52.788118 (+ 8869us) log.cc:842] Serialized 52 byte log entry
0806 03:43:53.760476 (+972358us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:54.111512 (+351036us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.194464 (+ 82952us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.199173 (+ 4709us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.212209 (+ 13036us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:54.212257 (+ 48us) raft_consensus.cc:1671] finished
0806 03:43:54.212268 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:54.212414 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":430172,"raft.queue_time_us":288322,"raft.run_cpu_time_us":514,"raft.run_wall_time_us":5279,"thread_start_us":1186113,"threads_started":2,"wal-append.queue_time_us":476538}
W20260806 03:43:54.254940 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 109041us
W20260806 03:43:54.259580 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 174990us
I20260806 03:43:54.261826 1670 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:54.212016 4114 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.269675 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:54.287010 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 116689us
I20260806 03:43:54.262149 4575 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:54.316279 1660 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.297276 4376 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:54.308424 501 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.292714 403 catalog_manager.cc:5495] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 4, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.265818 3902 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.343216 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.344319 498 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.328447 4431 consensus_queue.cc:1046] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260806 03:43:54.331540 4575 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:54.345585 4710 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
I20260806 03:43:54.346555 4607 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:54.347167 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.347746 1671 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.347823 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.348122 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.348683 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.349069 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.349367 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:54.349493 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 226834us
I20260806 03:43:54.349885 4663 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.350720 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:54.350965 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56674.7us
I20260806 03:43:54.351890 403 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:43:54.351660 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.351943 4473 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:43:54.354031 4736 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:54.328224 4376 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.365267 1658 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:54.344491 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1307) took 2755 ms. Trace:
I20260806 03:43:54.355392 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.368341 767 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:54.368980 4562 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:54.354468 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.375995 1670 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:54.370144 775 raft_consensus.cc:2354] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:54.342968 4422 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:43:54.371178 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 136568us
I20260806 03:43:54.356619 4566 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:54.383517 501 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:54.385219 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.385901 1658 raft_consensus.cc:2396] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 03:43:54.387321 4737 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:54.387881 768 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:43:54.388657 771 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:54.392985 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.367579 4626 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:54.393112 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 48972.2us
I20260806 03:43:54.375397 4558 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:54.394027 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.374774 546 rpcz_store.cc:276] 0806 03:43:51.584019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:53.605778 (+2021759us) service_pool.cc:227] Handling call
0806 03:43:53.688366 (+ 82588us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:53.834577 (+146211us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:53.834611 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:53.835371 (+ 760us) log.cc:842] Serialized 52 byte log entry
0806 03:43:54.117128 (+281757us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:54.332264 (+215136us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.338834 (+ 6570us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.339105 (+ 271us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.339173 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:54.339311 (+ 138us) raft_consensus.cc:1671] finished
0806 03:43:54.339324 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:54.339444 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36654,"prepare.queue_time_us":94253,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":39831,"raft.run_cpu_time_us":448,"raft.run_wall_time_us":7677,"thread_start_us":419576,"threads_started":2,"wal-append.queue_time_us":415551}
I20260806 03:43:54.401384 766 raft_consensus.cc:2379] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:54.397856 1670 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:54.411307 770 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:54.409245 616 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.378183 4607 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.420918 1668 raft_consensus.cc:1226] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:54.411976 4422 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:54.428077 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 51767.9us
I20260806 03:43:54.435983 4607 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.392470 4562 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.436942 4376 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.383932 4575 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.440368 1660 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:54.444902 1671 raft_consensus.cc:1261] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260806 03:43:54.445580 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 56693.4us
I20260806 03:43:54.449155 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:54.451200 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 370) took 1934 ms. Trace:
I20260806 03:43:54.467574 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.394778 4736 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.449975 4114 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:54.421065 4566 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.480376 4510 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.467022 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.486822 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.473078 4562 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:54.486075 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:54.437336 4626 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.455397 671 rpcz_store.cc:276] 0806 03:43:52.509145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:52.684071 (+174926us) service_pool.cc:227] Handling call
0806 03:43:52.700597 (+ 16526us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:52.711087 (+ 10490us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:43:53.746542 (+1035455us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.118491 (+371949us) log.cc:842] Serialized 14 byte log entry
0806 03:43:54.130989 (+ 12498us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:43:54.137512 (+ 6523us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:43:54.142149 (+ 4637us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:54.443733 (+301584us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b54000b2554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:54.443774 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:54.443886 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5824,"prepare.queue_time_us":126445,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":25773,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":65107,"spinlock_wait_cycles":29961856,"thread_start_us":1121636,"threads_started":2,"wal-append.queue_time_us":361391}
I20260806 03:43:54.462536 4547 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:54.500113 1671 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:43:54.494449 767 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
W20260806 03:43:54.506197 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 129339us
I20260806 03:43:54.482641 4024 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:43:54.511418 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 67101.1us
I20260806 03:43:54.505358 4547 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:54.486936 4431 consensus_queue.cc:1046] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260806 03:43:54.518980 1624 leader_election.cc:398] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:54.543367 1624 leader_election.cc:401] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 03:43:54.525699 4351 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:43:54.526834 1668 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:54.517915 4547 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:54.528934 4427 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.518791 4024 raft_consensus.cc:686] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:54.530469 3902 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.530352 728 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:54.532248 4626 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:54.535738 727 connection.cc:232] RPC call timeout handler was delayed by -1.1285340781323612s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260806 03:43:54.537698 4736 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:54.539170 4741 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260806 03:43:54.539484 4388 consensus_queue.cc:1046] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260806 03:43:54.541379 4575 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:54.522464 498 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:54.525146 4742 log.cc:924] Time spent T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260806 03:43:54.516595 1663 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:43:54.574344 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 44116.8us
I20260806 03:43:54.581981 728 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:54.580231 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.605469 4427 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:54.606570 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 120645us
W20260806 03:43:54.608247 4710 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.228s user 0.000s sys 0.002s
I20260806 03:43:54.544560 4351 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:54.609200 4351 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:54.614141 4566 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:54.623096 4268 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.585s user 0.009s sys 0.006s
I20260806 03:43:54.624028 1665 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:43:54.626724 4510 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:54.627384 4510 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:54.627885 4510 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:54.637406 4098 raft_consensus.cc:3032] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:54.645252 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 65716.7us
I20260806 03:43:54.638118 4098 raft_consensus.cc:729] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:54.652547 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 74865us
I20260806 03:43:54.645380 4636 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:54.664875 4689 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:43:54.670701 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 88807.9us
W20260806 03:43:54.674270 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 65802.6us
I20260806 03:43:54.678035 4464 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:54.678683 4750 log.cc:924] Time spent T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
W20260806 03:43:54.681183 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 158856us
I20260806 03:43:54.644421 4024 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.575325 4405 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.099s
I20260806 03:43:54.670679 4636 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:54.684000 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:54.730104 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 46678.7us
W20260806 03:43:54.836505 376 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 152557us
W20260806 03:43:54.729437 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 57709.1us
I20260806 03:43:54.655287 4688 raft_consensus.cc:3032] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:43:54.765228 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 83042.9us
W20260806 03:43:54.837476 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 138681us
W20260806 03:43:54.791850 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 280233us
I20260806 03:43:54.670806 4689 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.847075 1624 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:54.849495 4754 log.cc:924] Time spent T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 03:43:54.851459 4510 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.854121 4510 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.854554 4736 consensus_queue.cc:1046] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:54.697810 4607 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
I20260806 03:43:54.856272 4388 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:54.859992 4636 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:54.870743 4658 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:54.854892 3905 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:54.881587 4761 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:54.840521 4688 raft_consensus.cc:729] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:54.885739 1624 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:54.880419 616 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.865660 4689 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:43:54.884452 4388 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:54.686563 4098 consensus_queue.cc:260] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.730234 4451 consensus_queue.cc:1046] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.153s
I20260806 03:43:54.895994 4688 consensus_queue.cc:260] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.911600 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.913223 4688 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.901445 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.915436 4119 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:54.915853 4119 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:54.916175 4759 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:54.917428 4636 raft_consensus.cc:3032] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:54.917856 4636 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:54.913339 776 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:54.892854 4607 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:54.918834 4636 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.919476 4607 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.920188 4607 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:54.893906 4351 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.884848 4658 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.922149 4658 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.922894 4636 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.924825 4351 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.916224 4119 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:54.894675 4388 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.929292 4388 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:54.899942 3905 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.931388 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:54.932019 665 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:54.935166 665 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:54.935192 4497 raft_consensus.cc:3032] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:54.936420 4497 raft_consensus.cc:729] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:54.891386 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 205392us
I20260806 03:43:54.853914 4547 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.938899 615 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:54.940470 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 60175us
I20260806 03:43:54.941792 728 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:54.947319 4098 raft_consensus.cc:3037] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.947842 1665 raft_consensus.cc:1226] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:54.948020 4547 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.951900 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.948606 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:54.949296 615 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:54.953747 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:54.948547 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.957195 1657 raft_consensus.cc:2396] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 03:43:54.958791 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.939046 3905 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.960350 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:54.959574 776 raft_consensus.cc:2379] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:54.961037 4761 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:54.963999 1662 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:43:54.964293 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:54.950350 4351 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:54.971065 4351 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:54.971619 4351 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:54.937279 4497 consensus_queue.cc:260] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:54.981858 4119 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.982862 4658 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.985543 4658 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:54.985895 4761 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:54.987433 4761 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:54.988184 4119 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:54.996279 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:54.997150 769 raft_consensus.cc:1226] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:54.997270 770 raft_consensus.cc:2396] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:43:54.997712 4351 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.004046 4563 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:55.004559 4563 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.004676 773 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:55.004951 4563 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.007661 1668 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:55.013942 728 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:55.006606 3902 consensus_queue.cc:1035] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.010s
I20260806 03:43:55.018234 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:55.018724 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.018973 776 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:55.019294 771 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:55.020475 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.021293 769 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:55.022570 4636 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:55.023061 4636 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:55.023963 4636 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.028146 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:55.039719 776 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:54.991351 4497 raft_consensus.cc:3037] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:55.050974 767 raft_consensus.cc:1226] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:55.051266 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:55.048158 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.057379 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.058693 766 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:55.063247 4405 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.063426 4376 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.063545 4451 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.069708 1668 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:55.042676 4737 raft_consensus.cc:3032] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:43:54.992638 4547 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
W20260806 03:43:55.073870 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 73641.2us
W20260806 03:43:55.075753 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 48362.4us
I20260806 03:43:55.078526 616 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:55.078512 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:55.097638 1664 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:55.079939 771 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:55.064306 4405 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.104843 4405 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:55.074889 4737 raft_consensus.cc:729] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:55.102525 4689 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.100375 1624 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:55.108489 768 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:55.076323 4351 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.074738 4757 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:55.110963 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:55.128051 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 49660.5us
I20260806 03:43:55.128563 4737 consensus_queue.cc:260] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.129598 4772 consensus_queue.cc:1046] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:43:55.130074 4431 consensus_queue.cc:1035] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:43:55.129957 4737 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:55.137313 4626 consensus_queue.cc:1046] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:43:55.138433 4626 raft_consensus.cc:3032] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:55.138473 4119 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:55.138833 4626 raft_consensus.cc:729] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:55.139137 4119 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:43:55.139653 4119 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:55.140240 4575 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 03:43:55.140648 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.141163 4658 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:43:55.141548 4658 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:55.142506 4658 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.142940 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.143807 4658 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:43:55.144311 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.111783 4602 raft_consensus.cc:2781] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:55.145679 4602 raft_consensus.cc:686] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:55.145371 4626 consensus_queue.cc:260] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.111644 3902 raft_consensus.cc:2781] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:55.131600 4757 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.146458 4405 consensus_queue.cc:1046] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:55.144465 4759 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:55.105641 4510 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260806 03:43:55.147663 4405 raft_consensus.cc:3032] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:43:55.147385 4759 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.148072 4405 raft_consensus.cc:729] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:55.146864 4757 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:55.149011 4405 consensus_queue.cc:260] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.149679 4759 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:55.105727 4376 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.152115 4689 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.141566 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.150347 4405 raft_consensus.cc:3037] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:55.146700 4626 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:55.155577 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 44706.1us
I20260806 03:43:55.162855 4689 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.163578 4689 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.171078 4689 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.172206 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:55.173282 1664 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:55.174582 728 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:55.175604 4689 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:55.176031 4689 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.176388 4689 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:55.132109 769 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:55.186904 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.193025 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.193970 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.196784 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:55.198213 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:55.193320 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:55.200752 768 raft_consensus.cc:2396] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 03:43:55.184597 3902 raft_consensus.cc:686] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:55.201886 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.204358 564 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:43:55.205614 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:55.206615 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.207216 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.207913 4575 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:55.207895 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.208587 4575 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.208997 4575 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:55.209615 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.210772 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.188311 4431 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:55.211388 558 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:55.211381 4431 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.206635 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.212121 4431 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.195613 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:55.157042 4351 raft_consensus.cc:3032] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:55.210587 4566 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 03:43:55.202467 3902 consensus_queue.cc:237] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.212586 766 raft_consensus.cc:2362] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7. Re-sending same reply.
I20260806 03:43:55.219928 773 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:43:55.220616 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.152730 4451 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.224232 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.226011 768 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:55.152837 4602 consensus_queue.cc:237] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.199735 4427 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:55.231597 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 102729us
I20260806 03:43:55.218777 4351 raft_consensus.cc:729] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:55.227401 542 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:55.234203 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.237741 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.238301 558 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:55.240340 671 raft_consensus.cc:1226] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:55.241536 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:55.244889 4607 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.249646 4431 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.253391 4431 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:55.242183 1658 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:55.245131 4547 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:43:55.239282 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.267815 499 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:55.266049 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.279488 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:55.286347 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 50107.8us
I20260806 03:43:55.289960 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.230593 4376 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.295038 674 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:55.295135 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.296797 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.267076 4024 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.298497 664 raft_consensus.cc:1226] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:55.312570 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.300386 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.304186 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:55.299211 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.316067 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.317911 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.321235 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.298269 4351 consensus_queue.cc:260] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:55.321967 669 raft_consensus.cc:2354] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9.
I20260806 03:43:55.318804 546 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:55.317826 671 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:55.290300 3905 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:55.328934 768 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:43:55.340723 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.341390 546 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:55.287017 4427 consensus_queue.cc:1035] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260806 03:43:55.338114 3905 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.346299 4268 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:43:55.408746 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.418696 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.426008 663 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:55.309434 4563 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.398193 4024 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.564641 4575 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.486688 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.499573 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.406289 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:55.566689 1663 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:55.569372 4607 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:55.424135 4268 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:55.570765 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:55.571687 4782 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.253s user 0.006s sys 0.000s
I20260806 03:43:55.549206 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.573920 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.574785 663 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:55.425601 546 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:55.423688 3905 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.562976 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.565125 4024 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:55.486536 4781 log.cc:924] Time spent T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20260806 03:43:55.571103 4268 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.578490 558 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:55.550642 1657 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:55.588406 561 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
I20260806 03:43:55.624231 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.625360 671 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:55.575816 4098 consensus_meta.cc:287] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.613s user 0.015s sys 0.000s
I20260806 03:43:55.645769 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:55.646638 766 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.647850 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:55.651837 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.652514 558 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:55.654803 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.630085 564 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:43:55.641089 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.661247 661 raft_consensus.cc:2354] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
I20260806 03:43:55.664438 1660 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:55.587441 4761 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:55.671517 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.666998 499 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:55.658655 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.672724 663 raft_consensus.cc:2379] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:55.649739 4783 log.cc:924] Time spent T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.235s user 0.000s sys 0.005s
I20260806 03:43:55.615181 4688 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.678025 674 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:55.621832 4024 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.680918 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.682066 1657 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:55.692281 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.693418 564 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:55.587442 4119 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.712904 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.719009 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.617996 4689 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.720999 663 raft_consensus.cc:2354] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 8. Current term is 9.
I20260806 03:43:55.724068 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.681675 4473 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:55.685446 4575 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:55.682708 4388 consensus_queue.cc:1046] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260806 03:43:55.687793 4761 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.772601 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.795889 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.768692 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.758671 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.764122 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.838299 664 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:55.907971 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:55.885754 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 306712us
I20260806 03:43:55.763607 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.760625 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.771574 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.760813 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.758184 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.726303 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:55.910251 659 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
W20260806 03:43:55.892815 775 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.061s sys 0.098s
W20260806 03:43:55.911134 775 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.062s sys 0.098s
I20260806 03:43:55.757776 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.898572 766 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:55.916157 1624 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:55.919518 667 raft_consensus.cc:2354] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:43:55.926539 563 raft_consensus.cc:2362] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4. Re-sending same reply.
I20260806 03:43:55.716189 4451 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:55.768111 4405 consensus_meta.cc:287] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.583s user 0.009s sys 0.004s
W20260806 03:43:55.766076 4626 consensus_meta.cc:287] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.006s
W20260806 03:43:55.770815 4497 consensus_meta.cc:287] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.722s user 0.004s sys 0.012s
I20260806 03:43:55.760227 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.963435 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.928011 4811 raft_consensus.cc:2781] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:55.963435 4626 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:55.964947 4563 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.790470 4351 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.938992 1663 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:55.964581 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.965823 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.018116 671 raft_consensus.cc:2379] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:55.962692 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.955236 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:55.883288 4473 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:55.967748 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.019819 669 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:55.973291 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.696964 4688 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:55.974206 548 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:55.979084 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:55.983973 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:55.986646 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 113695us
I20260806 03:43:55.716976 4510 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.251s
W20260806 03:43:55.962215 4757 consensus_meta.cc:287] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.813s user 0.013s sys 0.000s
I20260806 03:43:55.989413 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:56.024255 674 raft_consensus.cc:2354] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:43:55.990459 544 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
W20260806 03:43:55.992566 4774 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.653s user 0.007s sys 0.000s
I20260806 03:43:55.991119 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:55.992820 4119 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:55.757261 4024 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.001740 766 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260806 03:43:55.911706 727 leader_election.cc:334] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.573s (SENT)
I20260806 03:43:55.955221 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.020542 564 raft_consensus.cc:2454] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:43:55.945443 4388 raft_consensus.cc:3032] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:56.028785 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:56.073572 614 leader_election.cc:334] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.545s (SENT)
W20260806 03:43:56.071944 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 160289us
I20260806 03:43:56.063352 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.107262 558 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:56.074827 615 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:56.075526 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.110805 561 raft_consensus.cc:2396] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:56.078490 659 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:43:56.069748 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.076146 546 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:56.074296 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.111028 616 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:56.030084 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 340789us
I20260806 03:43:56.082008 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.069479 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.063863 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.084679 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.031296 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.112766 663 raft_consensus.cc:2379] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:56.113564 667 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:56.069167 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.112521 671 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:56.108964 669 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:56.114446 547 raft_consensus.cc:2362] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5. Re-sending same reply.
I20260806 03:43:56.178663 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.179062 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.114156 1622 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:56.352360 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 141809us
I20260806 03:43:56.176038 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.174741 563 raft_consensus.cc:2454] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:56.179317 544 raft_consensus.cc:2362] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6. Re-sending same reply.
I20260806 03:43:56.065265 668 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:56.305419 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:56.239569 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.365020 1668 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:43:56.248842 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:56.247144 727 leader_election.cc:334] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.582s (SENT)
W20260806 03:43:56.248008 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 174449us
W20260806 03:43:56.083947 4768 log.cc:924] Time spent T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 03:43:56.367331 4405 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.022143 4473 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:56.367506 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:56.368387 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.366799 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.288277 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.361860 558 raft_consensus.cc:2362] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7. Re-sending same reply.
I20260806 03:43:56.378990 564 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
W20260806 03:43:56.350237 4810 log.cc:924] Time spent T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260806 03:43:56.272351 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.250422 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.175109 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.382495 547 raft_consensus.cc:2362] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7. Re-sending same reply.
I20260806 03:43:56.382177 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.292610 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:56.372632 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 258701us
I20260806 03:43:56.298659 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.333276 659 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 2.
I20260806 03:43:56.295827 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:56.387527 769 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:43:56.342548 661 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:56.376295 771 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:56.376703 727 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:56.283393 4807 log.cc:924] Time spent T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20260806 03:43:56.030478 4388 raft_consensus.cc:729] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:56.241700 4098 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:56.084406 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.248602 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:56.381278 1671 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:56.381330 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:56.396394 562 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
W20260806 03:43:56.181427 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 70484.5us
I20260806 03:43:56.383104 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.383656 563 raft_consensus.cc:2362] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7. Re-sending same reply.
I20260806 03:43:56.310565 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.072470 4788 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.024674 4811 raft_consensus.cc:686] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:56.384420 674 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:56.071439 4024 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:56.391355 614 leader_election.cc:334] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.565s (SENT)
I20260806 03:43:56.019382 4626 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.392359 663 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:56.404990 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.405822 547 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:56.376221 4798 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:56.392134 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.338039 4761 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:56.409036 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:56.083680 4809 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260806 03:43:56.085981 4689 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:56.180629 4431 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:56.279206 4268 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:56.394191 659 raft_consensus.cc:1261] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:43:56.353031 4566 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.063560 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:56.400378 616 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:56.401355 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.391522 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:56.459226 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.459669 667 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:56.432013 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.299474 4688 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:56.369961 4759 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.462385 4759 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.464509 4388 consensus_queue.cc:260] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.465793 4388 raft_consensus.cc:3037] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:56.403896 674 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:56.403610 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.408416 564 raft_consensus.cc:2396] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:43:56.467383 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.467665 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.410027 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:56.468843 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:56.469151 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 99091.8us
I20260806 03:43:56.469601 544 raft_consensus.cc:2379] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:56.469786 614 leader_election.cc:334] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.568s (SENT)
I20260806 03:43:56.409379 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.470201 614 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:56.471215 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.072118 4658 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.324s
I20260806 03:43:56.393236 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:56.472781 1660 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:43:56.472968 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 72649.2us
I20260806 03:43:56.474731 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:56.423959 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.476491 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.477066 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.477924 4351 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:56.401466 4829 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:56.478611 4351 raft_consensus.cc:2781] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:56.479132 4351 raft_consensus.cc:686] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:56.480177 4351 consensus_queue.cc:237] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.481352 4819 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:43:56.390071 4563 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.482849 4819 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:43:56.482671 4563 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.082887 3905 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.483780 728 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:56.484079 4626 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:56.389897 4405 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.423323 4497 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.485208 3905 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:56.485605 4497 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.486248 4759 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:56.472409 4811 consensus_queue.cc:237] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.487694 4497 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.410775 542 raft_consensus.cc:3037] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:56.461068 4566 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:56.462832 4813 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.145s user 0.001s sys 0.001s
W20260806 03:43:56.471364 614 leader_election.cc:334] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.606s (SENT)
I20260806 03:43:56.477226 548 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:56.031419 4757 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.472426 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.411731 561 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:56.491201 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.428989 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.493202 1668 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:56.511837 728 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:56.513674 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:56.515432 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 408635us
W20260806 03:43:56.431160 1622 leader_election.cc:334] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.085s (SENT)
I20260806 03:43:56.498890 616 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:56.501468 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1488) took 1237 ms. Trace:
I20260806 03:43:56.519284 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.521466 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.478613 4431 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:56.522662 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.523250 558 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:56.523170 4431 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.524192 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.524865 558 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:56.506022 4829 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:56.526909 4829 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:56.527372 4829 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:56.527281 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.528136 547 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:56.413475 4788 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.529006 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.533540 665 raft_consensus.cc:2354] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5.
I20260806 03:43:56.526938 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:56.525564 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.536849 547 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:56.537744 667 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:43:56.520781 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.539966 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:56.544124 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45348.4us
I20260806 03:43:56.518224 775 rpcz_store.cc:276] 0806 03:43:55.254678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:55.256178 (+ 1500us) service_pool.cc:227] Handling call
0806 03:43:55.307048 (+ 50870us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:55.307484 (+ 436us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:55.307533 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:55.315320 (+ 7787us) log.cc:842] Serialized 26 byte log entry
0806 03:43:55.911546 (+596226us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:55.911583 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:55.911647 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:56.486492 (+574845us) raft_consensus.cc:1671] finished
0806 03:43:56.486513 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:56.491928 (+ 5415us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5786,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11698,"thread_start_us":583253,"threads_started":1,"wal-append.queue_time_us":748283}
I20260806 03:43:56.546636 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.547811 1622 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:56.548846 4821 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:43:56.552014 727 leader_election.cc:334] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.613s (SENT)
I20260806 03:43:56.552642 727 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:56.554968 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.556478 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.559459 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.560317 548 raft_consensus.cc:2379] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 10: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:56.561386 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.562450 548 raft_consensus.cc:2396] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:43:56.563344 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.564270 548 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:56.566027 771 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:56.511499 4636 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.560230 558 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
W20260806 03:43:56.572319 665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2393) took 1433 ms (client timeout 1759 ms). Trace:
W20260806 03:43:56.576784 665 rpcz_store.cc:269] 0806 03:43:55.138781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:55.140341 (+ 1560us) service_pool.cc:227] Handling call
0806 03:43:56.572023 (+1431682us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540010f9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:56.572201 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3184557824}
I20260806 03:43:56.578665 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.579074 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:56.579566 544 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:56.580008 563 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:56.542938 4098 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:56.591022 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.592969 1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 473) took 1182 ms. Trace:
I20260806 03:43:56.549738 4759 raft_consensus.cc:2781] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:56.549427 4821 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.667917 558 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:56.822693 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:56.594521 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:56.870831 776 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
W20260806 03:43:56.650832 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 179480us
W20260806 03:43:56.868971 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 326985us
W20260806 03:43:56.872180 727 leader_election.cc:334] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.667s (SENT)
I20260806 03:43:56.872231 1624 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:56.870920 547 raft_consensus.cc:3037] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:56.875079 727 leader_election.cc:334] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.667s (SENT)
I20260806 03:43:56.875463 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:56.554543 4119 raft_consensus.cc:2781] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:43:56.660935 546 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:56.704258 564 raft_consensus.cc:2396] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260806 03:43:56.881294 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 450132us
W20260806 03:43:56.891690 614 leader_election.cc:334] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.619s (SENT)
W20260806 03:43:56.892786 614 leader_election.cc:334] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.683s (SENT)
W20260806 03:43:56.882473 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 308339us
I20260806 03:43:56.575798 4626 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:56.593484 1664 rpcz_store.cc:276] 0806 03:43:55.410392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:55.410543 (+ 151us) service_pool.cc:227] Handling call
0806 03:43:55.410887 (+ 344us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:55.411776 (+ 889us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:55.411809 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:55.499436 (+ 87627us) log.cc:842] Serialized 26 byte log entry
0806 03:43:55.673318 (+173882us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:43:55.673363 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:55.673430 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:56.368657 (+695227us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:56.586463 (+217806us) raft_consensus.cc:1671] finished
0806 03:43:56.586495 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:56.592860 (+ 6365us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":491,"raft.queue_time_us":87891,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":38136192,"thread_start_us":97690,"threads_started":1,"wal-append.queue_time_us":668434}
W20260806 03:43:56.844184 4830 log.cc:924] Time spent T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260806 03:43:56.900969 561 raft_consensus.cc:1226] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 6. Ops: []
I20260806 03:43:56.816284 4658 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:56.892227 1622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.116s (SENT) [suppressed 59 similar messages]
I20260806 03:43:56.757906 4431 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:56.858757 4788 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.778076 4566 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.863216 4563 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.701414 4800 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:56.826046 4405 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.603223 4759 raft_consensus.cc:686] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:56.700302 4798 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:56.741731 4689 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:56.742769 4388 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.850742 4757 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:56.927488 614 leader_election.cc:334] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.861s (SENT)
I20260806 03:43:56.768249 4451 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:56.894147 4819 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:56.955909 1664 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: []
W20260806 03:43:56.935382 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 367312us
I20260806 03:43:56.592264 4636 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.932585 4626 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:56.701200 4473 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.929131 4772 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:56.941743 1622 leader_election.cc:334] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.733s (SENT)
I20260806 03:43:56.969489 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:43:56.961426 614 leader_election.cc:334] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.891s (SENT)
I20260806 03:43:56.937209 4119 raft_consensus.cc:686] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:56.970443 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:56.950632 4388 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.985374 4119 consensus_queue.cc:237] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.986153 4388 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:56.986019 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:56.993256 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 118183us
I20260806 03:43:56.996078 562 raft_consensus.cc:2354] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6.
W20260806 03:43:56.997259 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 55549.6us
I20260806 03:43:56.996276 616 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:56.937951 4658 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.938964 4431 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.967154 1663 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:56.968145 1662 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:43:56.951042 4689 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:56.964474 4761 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
W20260806 03:43:57.017166 1622 leader_election.cc:334] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.210s (SENT)
W20260806 03:43:57.019773 727 connection.cc:232] RPC call timeout handler was delayed by -1.0643879389390349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
I20260806 03:43:56.966352 4451 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
W20260806 03:43:57.024571 727 connection.cc:232] RPC call timeout handler was delayed by -1.0238879390526563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
W20260806 03:43:57.026549 727 connection.cc:232] RPC call timeout handler was delayed by -1.0238879390526563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
W20260806 03:43:57.026889 727 connection.cc:232] RPC call timeout handler was delayed by -1.0220879390835762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21880000000000002s for a response
I20260806 03:43:57.024941 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:56.971341 4819 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:56.972699 4772 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:57.047915 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 51922.8us
I20260806 03:43:56.992664 4759 consensus_queue.cc:237] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:57.056327 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 60179.2us
I20260806 03:43:57.058566 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.024708 4832 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:43:57.062794 1671 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:57.063462 665 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:57.063519 661 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:57.082906 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.083751 659 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:57.052008 4800 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:57.086122 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.086894 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:57.052253 4839 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 03:43:57.066715 4829 consensus_meta.cc:287] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.539s user 0.013s sys 0.000s
I20260806 03:43:57.089733 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.090472 767 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
W20260806 03:43:57.078830 542 consensus_meta.cc:287] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.562s user 0.012s sys 0.000s
I20260806 03:43:57.094722 1622 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:57.098841 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:57.073993 4821 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:57.099474 770 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:57.108739 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 147344us
I20260806 03:43:57.099413 4811 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:43:57.121758 4689 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:57.099834 542 raft_consensus.cc:2454] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:43:57.122253 4473 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:57.122782 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1350) took 1045 ms. Trace:
I20260806 03:43:57.106415 4841 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:57.123831 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:57.123768 4841 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.124550 1664 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:57.112278 4566 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:43:57.134802 4566 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:43:57.135046 4844 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20260806 03:43:57.135485 4842 log.cc:924] Time spent T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260806 03:43:57.135962 4566 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.138716 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:57.140463 4837 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260806 03:43:57.140856 727 leader_election.cc:398] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
I20260806 03:43:57.146153 727 leader_election.cc:401] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:43:57.142081 4658 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:57.123136 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.112387 4861 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 03:43:57.094179 4829 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.143153 4451 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:57.143945 4772 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:57.144750 4835 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260806 03:43:57.094358 4839 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:57.184580 770 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:57.099792 4119 raft_consensus.cc:2781] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:57.156330 773 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:57.145351 4431 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:57.126034 4833 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260806 03:43:57.165498 4689 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
I20260806 03:43:57.165553 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:57.172688 1664 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:57.173370 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.203647 499 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:57.178224 4268 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.682s user 0.007s sys 0.006s
I20260806 03:43:57.211994 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 379) took 1423 ms. Trace:
I20260806 03:43:57.214558 775 raft_consensus.cc:2396] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:43:57.213882 669 rpcz_store.cc:276] 0806 03:43:55.788641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.389972 (+601331us) service_pool.cc:227] Handling call
0806 03:43:56.393366 (+ 3394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:56.409155 (+ 15789us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:56.409185 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:56.432363 (+ 23178us) log.cc:842] Serialized 26 byte log entry
0806 03:43:56.735461 (+303098us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:56.735500 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:56.735581 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:57.211764 (+476183us) raft_consensus.cc:1671] finished
0806 03:43:57.211781 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:57.211902 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11935,"raft.queue_time_us":10932,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":11235,"thread_start_us":225801,"threads_started":1,"wal-append.queue_time_us":533982}
I20260806 03:43:57.213889 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:57.208249 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 113603us
W20260806 03:43:57.216342 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 50925.3us
I20260806 03:43:57.218034 727 leader_election.cc:398] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:43:57.234164 4119 raft_consensus.cc:686] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:57.234771 4636 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:57.235365 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:57.219000 4473 raft_consensus.cc:2781] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:43:57.123179 542 rpcz_store.cc:276] 0806 03:43:56.077190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.084110 (+ 6920us) service_pool.cc:227] Handling call
0806 03:43:57.122708 (+1038598us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:43:57.216023 4851 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:57.252503 4821 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:57.252867 4761 consensus_queue.cc:1046] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:43:57.255154 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 643) took 1168 ms. Trace:
I20260806 03:43:57.211822 4841 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:57.242550 4689 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:57.298834 547 raft_consensus.cc:2454] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:43:57.211710 4829 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:57.210079 4819 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:57.324071 1622 leader_election.cc:334] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.209s (SENT)
I20260806 03:43:57.331722 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:57.334856 4689 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:57.328840 1658 rpcz_store.cc:276] 0806 03:43:56.086915 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.087046 (+ 131us) service_pool.cc:227] Handling call
0806 03:43:56.087408 (+ 362us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:56.335973 (+248565us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:56.336054 (+ 81us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:56.363867 (+ 27813us) log.cc:842] Serialized 26 byte log entry
0806 03:43:56.655981 (+292114us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:43:56.656033 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:56.656133 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:57.254946 (+598813us) raft_consensus.cc:1671] finished
0806 03:43:57.254964 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:57.255095 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":236164,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":287844,"threads_started":1,"wal-append.queue_time_us":561190}
I20260806 03:43:57.322474 4473 raft_consensus.cc:686] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:57.369827 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.370944 659 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:57.377521 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.378641 663 raft_consensus.cc:2379] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:57.355818 4689 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:57.383026 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.382841 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.383934 661 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:57.401813 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.401278 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.411366 667 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:57.411948 665 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:43:57.385401 771 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 03:43:57.401444 669 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 9. Ops: []
I20260806 03:43:57.367681 4658 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:43:57.402349 4119 consensus_queue.cc:237] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:57.422243 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 312877us
I20260806 03:43:57.334722 4821 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:57.433265 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1360) took 1099 ms. Trace:
W20260806 03:43:57.434437 614 leader_election.cc:334] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.045s (SENT)
I20260806 03:43:57.430490 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.435228 614 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:57.434340 563 rpcz_store.cc:276] 0806 03:43:56.334254 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.580766 (+246512us) service_pool.cc:227] Handling call
0806 03:43:56.581232 (+ 466us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:56.595033 (+ 13801us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:56.595074 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:56.892309 (+297235us) log.cc:842] Serialized 52 byte log entry
0806 03:43:57.010734 (+118425us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:43:57.073000 (+ 62266us) log.cc:842] Serialized 14 byte log entry
0806 03:43:57.077715 (+ 4715us) log.cc:842] Serialized 14 byte log entry
0806 03:43:57.077966 (+ 251us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:57.078057 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:57.403582 (+325525us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b18001aa2e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:43:57.432987 (+ 29405us) raft_consensus.cc:1671] finished
0806 03:43:57.433031 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:57.433196 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4364,"prepare.queue_time_us":204172,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":20030,"raft.queue_time_us":13579,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":61869,"spinlock_wait_cycles":16923776,"thread_start_us":117389,"threads_started":2,"wal-append.queue_time_us":137642}
W20260806 03:43:57.457010 4837 log.cc:924] Time spent T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260806 03:43:57.402091 4862 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260806 03:43:57.328207 4851 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:57.429962 4867 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 03:43:57.468245 4819 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:57.463549 4268 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:57.526764 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 202722us
W20260806 03:43:57.532863 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 247333us
I20260806 03:43:57.540294 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:57.539019 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:57.647022 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:57.648941 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 214548us
W20260806 03:43:57.653270 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 113081us
I20260806 03:43:57.653911 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.656437 769 raft_consensus.cc:2396] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:43:57.658064 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.658293 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.659394 773 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:43:57.658131 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:57.646292 4867 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260806 03:43:57.666174 1670 raft_consensus.cc:2379] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:43:57.670107 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 131203us
I20260806 03:43:57.674299 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.675215 771 raft_consensus.cc:2396] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:57.525447 4268 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.651201 4871 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 03:43:57.668965 4626 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:57.665303 4405 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:57.665660 4388 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:57.651648 4510 consensus_queue.cc:1046] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260806 03:43:56.240876 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:43:57.693418 403 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51378 (request call id 179) took 14844 ms (client timeout 15000 ms). Trace:
I20260806 03:43:57.693892 616 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:57.805341 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 111526us
I20260806 03:43:57.806288 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:57.807718 616 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:57.687451 614 leader_election.cc:334] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.111s (SENT)
W20260806 03:43:57.809043 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 121599us
W20260806 03:43:57.809806 614 leader_election.cc:334] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.119s (SENT)
I20260806 03:43:57.810250 614 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:57.679766 4821 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:43:57.786092 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 113818us
I20260806 03:43:57.673367 4098 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:57.699929 4626 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:43:57.860172 767 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.024s sys 0.063s
W20260806 03:43:57.787781 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 56380.1us
I20260806 03:43:57.813146 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:57.536768 4473 consensus_queue.cc:237] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.858099 4850 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:57.676267 4829 consensus_queue.cc:1035] T 68a9e5190380411684128482e3a899b8 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260806 03:43:57.825186 4772 raft_consensus.cc:2781] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:43:57.802724 403 rpcz_store.cc:269] 0806 03:43:42.849729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:42.850123 (+ 394us) service_pool.cc:227] Handling call
0806 03:43:57.693326 (+14843203us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:43:54.678145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:43:54.678891 (+ 746us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:43:54.678992 (+ 101us) write_op.cc:435] Acquired schema lock
0806 03:43:54.679013 (+ 21us) tablet.cc:660] Decoding operations
0806 03:43:54.680145 (+ 1132us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:43:54.680265 (+ 120us) write_op.cc:641] Partition lock acquired for write op
0806 03:43:54.680291 (+ 26us) tablet.cc:683] Acquiring locks for 11 operations
0806 03:43:54.813135 (+132844us) tablet.cc:699] Row locks acquired
0806 03:43:54.813197 (+ 62us) write_op.cc:260] PREPARE: finished
0806 03:43:54.825097 (+ 11900us) write_op.cc:270] Start()
0806 03:43:54.825362 (+ 265us) write_op.cc:276] Timestamp: P: 1785987834825016 usec, L: 0
0806 03:43:54.825421 (+ 59us) op_driver.cc:331] REPLICATION: starting
0806 03:43:54.828211 (+ 2790us) log.cc:842] Serialized 4367 byte log entry
0806 03:43:55.203424 (+375213us) op_driver.cc:444] REPLICATION: finished
0806 03:43:55.408600 (+205176us) write_op.cc:301] APPLY: starting
0806 03:43:55.462051 (+ 53451us) tablet.cc:1365] starting BulkCheckPresence
0806 03:43:55.487900 (+ 25849us) tablet.cc:1368] finished BulkCheckPresence
0806 03:43:55.621628 (+133728us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:43:55.751307 (+129679us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:43:55.751782 (+ 475us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0806 03:43:55.751852 (+ 70us) write_op.cc:312] APPLY: finished
0806 03:43:55.752398 (+ 546us) log.cc:842] Serialized 83 byte log entry
0806 03:43:55.752641 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:55.752848 (+ 207us) write_op.cc:454] Released schema lock
0806 03:43:55.752954 (+ 106us) write_op.cc:341] FINISH: Updating metrics
0806 03:43:55.753347 (+ 393us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:43:55.753373 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":560038,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":144182,"apply.run_cpu_time_us":10352,"apply.run_wall_time_us":422257,"num_ops":11,"prepare.queue_time_us":196036,"prepare.run_cpu_time_us":15499,"prepare.run_wall_time_us":231242,"raft.queue_time_us":209983,"raft.run_cpu_time_us":50643,"raft.run_wall_time_us":127376,"replication_time_us":374468,"thread_start_us":425409,"threads_started":4,"wal-append.queue_time_us":121102,"wal-append.run_cpu_time_us":48310,"wal-append.run_wall_time_us":1808127}]]}
I20260806 03:43:57.860168 4819 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:43:57.858582 4510 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 03:43:57.891594 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 78647.6us
I20260806 03:43:57.866777 4626 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:57.787820 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 87186.7us
W20260806 03:43:57.865274 767 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s user 0.025s sys 0.063s
W20260806 03:43:57.890187 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 66791us
I20260806 03:43:57.864511 4405 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.824625 4098 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:57.896340 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:57.825634 4388 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:57.867556 4863 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:57.851933 4821 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
W20260806 03:43:57.925498 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 115728us
W20260806 03:43:57.926506 614 leader_election.cc:334] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.138s (SENT)
W20260806 03:43:57.869882 1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 477) took 1870 ms (client timeout 1702 ms). Trace:
I20260806 03:43:57.892642 4819 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:57.932410 4845 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:57.932739 4819 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:57.932847 4388 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:57.913249 4863 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:57.933836 4863 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:57.934831 4636 raft_consensus.cc:3032] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:57.935248 4636 raft_consensus.cc:729] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
W20260806 03:43:57.927598 614 leader_election.cc:334] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.161s (SENT)
I20260806 03:43:57.910439 4098 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:43:57.949905 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 79911.9us
I20260806 03:43:57.924957 4850 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:57.935738 4821 consensus_queue.cc:1225] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260806 03:43:57.932891 4845 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:43:57.977164 563 raft_consensus.cc:3032] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:57.980963 563 raft_consensus.cc:729] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:57.982046 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 85821.3us
W20260806 03:43:57.989313 727 leader_election.cc:334] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.546s (SENT)
I20260806 03:43:57.992873 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:57.989832 727 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:57.930260 1662 rpcz_store.cc:269] 0806 03:43:55.998335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:55.999747 (+ 1412us) service_pool.cc:227] Handling call
0806 03:43:57.860437 (+1860690us) spinlock_profiling.cc:243] Waited 72 ms on lock 0x7b5400642654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:57.868199 (+ 7762us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165591296}
I20260806 03:43:57.993547 776 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:58.006093 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:58.015758 4888 log.cc:924] Time spent T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20260806 03:43:58.016247 4890 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260806 03:43:58.018618 4891 log.cc:924] Time spent T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.068s user 0.000s sys 0.002s
W20260806 03:43:58.022751 4873 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.228s user 0.000s sys 0.000s
I20260806 03:43:58.024315 4851 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.024969 4658 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:43:58.025394 4658 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:58.025758 4658 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:58.026201 4832 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:58.026638 4832 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:43:58.032703 562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 399) took 1690 ms (client timeout 2085 ms). Trace:
W20260806 03:43:58.033141 562 rpcz_store.cc:269] 0806 03:43:56.342036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.476750 (+134714us) service_pool.cc:227] Handling call
0806 03:43:58.032467 (+1555717us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54002a0654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:58.032632 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1185480960}
I20260806 03:43:58.034093 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.034958 674 raft_consensus.cc:2379] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:57.985064 4829 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:43:58.036111 614 leader_election.cc:334] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.228s (SENT)
I20260806 03:43:57.978585 4268 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:58.039553 4829 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:58.063063 4829 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:58.065516 564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 403) took 1605 ms (client timeout 1693 ms). Trace:
I20260806 03:43:58.042436 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:58.065940 564 rpcz_store.cc:269] 0806 03:43:56.460695 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.467441 (+ 6746us) service_pool.cc:227] Handling call
0806 03:43:58.065306 (+1597865us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540011dfd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:58.065454 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541795712}
I20260806 03:43:58.042271 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.045275 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:58.067056 668 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:43:58.067425 664 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:58.043000 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:58.049650 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 467595us
I20260806 03:43:58.044984 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.060218 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.067018 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.060323 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.059461 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:58.058674 546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1351) took 1981 ms (client timeout 2228 ms). Trace:
I20260806 03:43:58.071077 561 raft_consensus.cc:3037] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:58.059104 548 raft_consensus.cc:1261] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:43:58.059859 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:58.071255 546 rpcz_store.cc:269] 0806 03:43:56.077442 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.402228 (+324786us) service_pool.cc:227] Handling call
0806 03:43:58.058496 (+1656268us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54002b0f54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:58.058618 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101607168}
I20260806 03:43:58.072252 558 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 2.
I20260806 03:43:58.054509 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 879) took 1658 ms. Trace:
I20260806 03:43:58.058743 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:58.042249 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.044662 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.068122 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.045687 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:58.042706 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:57.976935 4772 raft_consensus.cc:686] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:58.070184 563 consensus_queue.cc:260] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.035262 4845 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:57.985245 4636 consensus_queue.cc:260] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.072947 1671 rpcz_store.cc:276] 0806 03:43:56.396197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.404220 (+ 8023us) service_pool.cc:227] Handling call
0806 03:43:56.427992 (+ 23772us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:57.063260 (+635268us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:58.054176 (+990916us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54007030d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:58.054439 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1459384576}
I20260806 03:43:58.177443 562 raft_consensus.cc:3032] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:58.409426 562 raft_consensus.cc:729] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:58.048346 4658 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.412252 1662 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:58.413574 771 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260806 03:43:58.412171 4819 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.416002 4098 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.415460 4626 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:58.429818 4899 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.497s user 0.001s sys 0.007s
I20260806 03:43:58.132942 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.060686 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.145066 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.409533 563 raft_consensus.cc:3037] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:58.426901 4845 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:43:58.410235 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 374152us
I20260806 03:43:58.248190 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.456513 558 raft_consensus.cc:2396] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:43:58.458117 776 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:43:58.457376 769 raft_consensus.cc:2396] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260806 03:43:58.460417 665 raft_consensus.cc:2354] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 8. Current term is 9.
I20260806 03:43:58.143980 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.461488 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.462023 544 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:58.462477 659 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:43:58.174247 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 184948us
W20260806 03:43:58.144021 1622 leader_election.cc:334] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 3.147s (SENT)
I20260806 03:43:58.351315 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57222 (request call id 1504) took 1229 ms. Trace:
I20260806 03:43:58.464529 775 rpcz_store.cc:276] 0806 03:43:57.121861 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:57.125465 (+ 3604us) service_pool.cc:227] Handling call
0806 03:43:58.340320 (+1214855us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5400312754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:58.351205 (+ 10885us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25521408}
I20260806 03:43:58.418507 4863 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.468313 4658 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:58.470139 667 raft_consensus.cc:2354] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:43:58.473562 4821 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:58.474272 4821 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.474949 4497 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:43:58.455201 614 leader_election.cc:334] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.229s (SENT)
I20260806 03:43:58.477803 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.484984 1657 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 03:43:58.369270 4689 consensus_meta.cc:287] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.914s user 0.005s sys 0.005s
I20260806 03:43:58.485739 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.478519 661 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:43:58.487699 1622 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:58.496976 773 raft_consensus.cc:2379] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:58.503576 769 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260806 03:43:58.488269 727 leader_election.cc:334] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.551s (SENT)
W20260806 03:43:58.056563 4836 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 1.089s user 0.000s sys 0.003s
I20260806 03:43:58.507983 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:58.394642 4911 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:58.391420 4405 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.518770 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.516511 669 raft_consensus.cc:1226] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260806 03:43:58.514228 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.520236 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.521054 767 raft_consensus.cc:1226] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 3. Current term is 4. Ops: []
I20260806 03:43:58.059145 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:58.532138 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 76992.7us
W20260806 03:43:58.568869 614 leader_election.cc:334] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.477s (SENT)
I20260806 03:43:58.476473 4636 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:58.098940 4510 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:58.572453 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:58.575838 767 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 9
I20260806 03:43:58.520217 768 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:43:58.531315 773 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260806 03:43:58.566910 1658 raft_consensus.cc:1226] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:58.581051 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.582115 772 raft_consensus.cc:2396] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:43:58.518474 4268 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:58.587399 775 raft_consensus.cc:1226] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:58.249049 4451 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:58.567715 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:43:58.593786 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 523826us
I20260806 03:43:58.603845 4821 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.531914 4626 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.604887 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:58.605921 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.475863 4497 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.531342 4911 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.586522 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:58.621994 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 158247us
I20260806 03:43:58.672983 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:43:58.673527 727 leader_election.cc:334] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.702s (SENT)
I20260806 03:43:58.673911 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:58.510982 4689 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.624617 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1346) took 2548 ms. Trace:
I20260806 03:43:58.424048 4772 consensus_queue.cc:237] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.566295 4757 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260806 03:43:58.574476 4914 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260806 03:43:58.608765 4871 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:58.530961 4878 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
W20260806 03:43:58.653862 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 85037.9us
I20260806 03:43:58.658432 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.689224 768 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 2.
I20260806 03:43:58.656574 661 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:43:58.651249 558 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:43:58.674789 767 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:43:58.693468 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:58.666563 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:58.653409 4832 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.626s user 0.009s sys 0.000s
I20260806 03:43:58.608515 4510 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:58.674270 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 401) took 1296 ms. Trace:
I20260806 03:43:58.673928 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:58.677011 548 rpcz_store.cc:276] 0806 03:43:56.075932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:56.571986 (+496054us) service_pool.cc:227] Handling call
0806 03:43:58.058648 (+1486662us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:58.071876 (+ 13228us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:58.624381 (+552505us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002b0f50. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:58.624549 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3418126720}
W20260806 03:43:58.697189 727 leader_election.cc:334] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.751s (SENT)
I20260806 03:43:58.699100 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:58.693216 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:58.700196 501 connection.cc:232] RPC call timeout handler was delayed by -1.7664196928963065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260806 03:43:58.691239 668 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:43:58.691850 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:43:58.700676 501 connection.cc:232] RPC call timeout handler was delayed by -1.7628196929581463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260806 03:43:58.698568 1671 raft_consensus.cc:2396] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:43:58.680949 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:43:58.701004 501 connection.cc:232] RPC call timeout handler was delayed by -1.5162196930032223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
W20260806 03:43:58.701293 501 connection.cc:232] RPC call timeout handler was delayed by -1.5045196928549558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260806 03:43:58.701380 769 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:43:58.665078 561 consensus_meta.cc:287] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.514s user 0.008s sys 0.003s
I20260806 03:43:58.702003 561 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:58.700281 772 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:43:58.701588 501 connection.cc:232] RPC call timeout handler was delayed by -1.466719692805782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
W20260806 03:43:58.702874 501 connection.cc:232] RPC call timeout handler was delayed by -1.460419693030417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.189s for a response
W20260806 03:43:58.703207 501 connection.cc:232] RPC call timeout handler was delayed by -1.2156196928117424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260806 03:43:58.703485 501 connection.cc:232] RPC call timeout handler was delayed by -1.1616196928080171s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260806 03:43:58.703979 501 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:58.694151 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:58.698714 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:58.712841 661 raft_consensus.cc:2379] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:58.712049 1657 raft_consensus.cc:2354] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
I20260806 03:43:58.716187 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:58.716936 561 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:58.686069 4861 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:58.609248 4451 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.722393 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:43:58.724263 728 leader_election.cc:398] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
I20260806 03:43:58.724732 728 leader_election.cc:401] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 03:43:58.725049 547 raft_consensus.cc:2454] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:58.726032 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:58.726752 768 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:58.696022 671 rpcz_store.cc:276] 0806 03:43:57.377389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:57.378493 (+ 1104us) service_pool.cc:227] Handling call
0806 03:43:57.650787 (+272294us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 03:43:57.663047 (+ 12260us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:57.663082 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 03:43:57.677003 (+ 13921us) log.cc:842] Serialized 78 byte log entry
0806 03:43:57.870282 (+193279us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:57.870862 (+ 580us) log.cc:842] Serialized 14 byte log entry
0806 03:43:57.893107 (+ 22245us) log.cc:842] Serialized 14 byte log entry
0806 03:43:57.946601 (+ 53494us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:57.947373 (+ 772us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:43:58.429162 (+481789us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:43:58.674043 (+244881us) raft_consensus.cc:1671] finished
0806 03:43:58.674069 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:43:58.674213 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32263,"prepare.queue_time_us":246401,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":3888,"raft.queue_time_us":126533,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":3794,"spinlock_wait_cycles":63438720,"thread_start_us":125378,"threads_started":2,"wal-append.queue_time_us":239588}
I20260806 03:43:58.654937 669 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb"
dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
mode: GRACEFUL
new_leader_uuid: "ada5ef6656cb4420baf77a4b4342909e"
from {username='slave'} at 127.0.0.1:53596
W20260806 03:43:58.686108 4119 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.024s sys 0.045s
W20260806 03:43:58.718356 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 44477.4us
I20260806 03:43:58.687608 1662 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:43:58.719388 767 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:58.729276 669 raft_consensus.cc:595] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Received request to transfer leadership to TS ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:58.732333 727 leader_election.cc:334] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.834s (SENT)
I20260806 03:43:58.737094 669 raft_consensus.cc:603] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:43:58.728642 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:58.740674 1622 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:58.712590 4906 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:58.747053 1663 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:58.687292 4405 consensus_queue.cc:1046] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260806 03:43:58.706538 4916 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:58.743795 727 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:58.743116 542 raft_consensus.cc:1261] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:58.690510 4871 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:58.747193 4829 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.684s user 0.010s sys 0.000s
I20260806 03:43:58.743453 661 raft_consensus.cc:2379] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:58.693215 4788 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
W20260806 03:43:58.736510 4119 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.024s sys 0.045s
I20260806 03:43:58.726248 4268 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:58.725710 4861 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:58.728410 4473 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:58.741050 4658 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:43:58.770300 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.772049 768 raft_consensus.cc:2454] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:43:58.768414 561 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:43:58.768764 769 raft_consensus.cc:2454] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:58.775199 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:58.763306 4917 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260806 03:43:58.776695 548 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:43:58.793778 616 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:58.794984 616 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:58.786664 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.712138 4832 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.798678 546 raft_consensus.cc:2454] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:58.773874 4861 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:58.769436 4268 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:58.773206 4916 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:58.773481 4473 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:58.814213 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 121427us
I20260806 03:43:58.817515 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:58.800007 4900 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.021s sys 0.014s
I20260806 03:43:58.827904 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.828791 548 raft_consensus.cc:2379] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:43:58.829118 4918 log.cc:924] Time spent T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20260806 03:43:58.831459 1670 raft_consensus.cc:2354] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for earlier term 8. Current term is 9.
I20260806 03:43:58.831753 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.832365 561 raft_consensus.cc:2362] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6. Re-sending same reply.
I20260806 03:43:58.758106 4829 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.808868 4119 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:43:58.818908 4841 raft_consensus.cc:2781] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:43:58.818512 4268 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:58.817132 4911 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:58.850633 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 118317us
W20260806 03:43:58.858171 727 leader_election.cc:334] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.951s (SENT)
I20260806 03:43:58.858736 727 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:58.823508 4900 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.021s sys 0.014s
W20260806 03:43:58.852203 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 121127us
I20260806 03:43:58.824944 4473 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:58.863690 776 raft_consensus.cc:1226] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:58.763604 4906 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:43:58.866474 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 172401us
W20260806 03:43:58.861203 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 118880us
I20260806 03:43:58.847450 4497 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.746348 4915 consensus_queue.cc:1046] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
W20260806 03:43:58.869055 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 116901us
I20260806 03:43:58.880288 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:58.867787 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:43:58.871771 614 leader_election.cc:334] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.652s (SENT)
I20260806 03:43:58.866636 4841 raft_consensus.cc:686] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:58.875973 4832 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:58.876585 4626 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:58.881407 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:58.883566 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.887745 771 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:43:58.884429 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.857107 4119 raft_consensus.cc:686] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:58.896021 1657 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:43:58.895152 564 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:43:58.896118 770 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:43:58.897096 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.896736 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.898110 548 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 10.
I20260806 03:43:58.889726 563 raft_consensus.cc:1261] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 03:43:58.899497 563 pending_rounds.cc:76] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 3
I20260806 03:43:58.899360 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:58.899874 563 pending_rounds.cc:97] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 03:43:58.900395 563 raft_consensus.cc:2866] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:58.906119 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.908775 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:58.872435 4871 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:58.909376 771 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:58.916275 616 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:58.882117 4689 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:58.877753 4926 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:43:58.929363 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 57639.7us
W20260806 03:43:58.933286 614 leader_election.cc:334] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 3.207s (SENT)
I20260806 03:43:58.933701 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:58.929333 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 49111.4us
I20260806 03:43:58.935495 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:58.936689 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61177.8us
W20260806 03:43:58.937589 727 leader_election.cc:334] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.023s (SENT)
I20260806 03:43:58.948575 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.917171 4903 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:43:58.951820 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:58.931746 562 consensus_queue.cc:260] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:58.960964 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:58.961561 562 raft_consensus.cc:3037] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:58.961635 548 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:58.936451 4757 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:58.937507 4928 raft_consensus.cc:3032] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:58.938035 4451 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:58.933523 4658 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:43:58.951627 4905 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.008s user 0.013s sys 0.011s
I20260806 03:43:58.902493 4832 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260806 03:43:58.944108 4911 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:43:58.938402 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:58.970805 4757 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:43:58.982529 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 44956.6us
I20260806 03:43:58.970610 4689 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:43:59.000626 727 leader_election.cc:334] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.112s (SENT)
I20260806 03:43:58.982550 4932 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:43:58.983024 4933 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:58.983172 4927 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:58.931597 4405 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260806 03:43:58.984169 4658 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.991204 771 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:43:58.970741 4903 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:58.979702 4905 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.048s user 0.014s sys 0.011s
I20260806 03:43:59.018941 772 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:43:58.995460 4906 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.003197 4757 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:58.965104 548 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:43:58.954327 4878 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260806 03:43:59.002718 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:43:59.030110 561 raft_consensus.cc:2362] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8. Re-sending same reply.
I20260806 03:43:58.973944 4772 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.007092 770 raft_consensus.cc:1261] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 03:43:59.013576 563 raft_consensus.cc:1261] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 03:43:58.972018 4841 consensus_queue.cc:237] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.003764 4932 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.013144 4927 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:58.970876 4928 raft_consensus.cc:729] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.020360 616 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:58.998234 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:43:59.020664 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 148009us
I20260806 03:43:59.044404 544 raft_consensus.cc:2362] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6. Re-sending same reply.
W20260806 03:43:59.045723 1622 leader_election.cc:334] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.503s (SENT)
I20260806 03:43:59.046164 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:59.047303 1622 leader_election.cc:334] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.554s (SENT)
I20260806 03:43:59.008311 4119 consensus_queue.cc:237] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.022758 4510 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 7. Current term is 8.
W20260806 03:43:58.962680 4497 consensus_queue.cc:1159] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" responder_term: 7 status { last_received { term: 6 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:43:59.012648 4933 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.045543 4932 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:59.050985 4928 consensus_queue.cc:260] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.052353 4928 raft_consensus.cc:3037] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.053573 4473 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.057727 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:59.059077 1622 leader_election.cc:334] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.693s (SENT)
W20260806 03:43:59.059927 1622 leader_election.cc:334] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.736s (SENT)
I20260806 03:43:59.060307 1622 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:59.017997 1624 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:43:59.029462 727 leader_election.cc:334] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.216s (SENT)
I20260806 03:43:59.062744 727 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.008867 4829 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:59.026093 4903 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.042022 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.067656 562 raft_consensus.cc:1261] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260806 03:43:59.037038 614 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:59.059268 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.068317 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1377) took 1011 ms. Trace:
I20260806 03:43:59.068698 562 rpcz_store.cc:276] 0806 03:43:58.057252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:58.096449 (+ 39197us) service_pool.cc:227] Handling call
0806 03:43:58.177280 (+ 80831us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:58.891261 (+713981us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b4c00240d80. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32146e2210 00007f32146f2fb2 00007f32146b9ac5 00007f32146e071d 00007f32146e0383 00007f321631a4fb 00007f321631b025 00007f32163162ef 00007f3216317b4e 00007f3216321c89 00007f3216322556
0806 03:43:59.068056 (+176795us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:59.068260 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4125568}
I20260806 03:43:59.067572 667 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:59.078109 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.079407 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:59.094655 498 connection.cc:232] RPC call timeout handler was delayed by -2.8388111318927258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20260806 03:43:59.097357 498 connection.cc:232] RPC call timeout handler was delayed by -2.8352111319545656s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
W20260806 03:43:59.099411 498 connection.cc:232] RPC call timeout handler was delayed by -2.6507111319806427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260806 03:43:59.101022 498 connection.cc:232] RPC call timeout handler was delayed by -2.5886111319996417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260806 03:43:59.078902 659 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:59.079041 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61199.8us
W20260806 03:43:59.106920 498 connection.cc:232] RPC call timeout handler was delayed by -2.5769111318513751s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
W20260806 03:43:59.107188 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 80243.3us
I20260806 03:43:59.108989 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:59.109637 667 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:43:59.067135 4688 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:43:59.103691 4626 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:59.111830 4688 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:59.112726 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.112864 4688 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.113458 4878 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:59.113865 4878 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.114195 4878 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.115343 4935 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.115834 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.116952 4943 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:43:59.117007 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.117810 4943 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:43:59.114020 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.029208 4915 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260806 03:43:59.139990 558 raft_consensus.cc:1261] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:43:59.119405 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.126734 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:59.146569 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 123003us
W20260806 03:43:59.147480 727 leader_election.cc:334] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.231s (SENT)
I20260806 03:43:59.118304 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:59.153934 498 connection.cc:232] RPC call timeout handler was delayed by -2.5328111320268363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.189s for a response
W20260806 03:43:59.158092 498 connection.cc:232] RPC call timeout handler was delayed by -2.4770111318212003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260806 03:43:59.153872 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:59.162379 664 raft_consensus.cc:3037] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:43:59.161648 498 connection.cc:232] RPC call timeout handler was delayed by -2.4167111318092793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
W20260806 03:43:59.166028 498 connection.cc:232] RPC call timeout handler was delayed by -2.3240111318882555s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2122s for a response
I20260806 03:43:59.166852 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:59.169001 1664 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:59.169353 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:43:59.168216 498 connection.cc:232] RPC call timeout handler was delayed by -2.2880111318081617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2162s for a response
W20260806 03:43:59.130458 4845 consensus_meta.cc:287] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.628s user 0.010s sys 0.002s
I20260806 03:43:59.178763 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.126006 3902 raft_consensus.cc:3032] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:59.180339 667 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:43:59.118264 4943 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:43:59.182107 659 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:43:59.172192 498 connection.cc:232] RPC call timeout handler was delayed by -1.6724111319053918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2846s for a response
I20260806 03:43:59.042361 4772 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.184680 766 raft_consensus.cc:1261] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 03:43:59.191116 766 pending_rounds.cc:76] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 7
I20260806 03:43:59.191418 766 pending_rounds.cc:97] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260806 03:43:59.168663 4907 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.168798 4098 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:59.169276 4829 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:43:59.193250 727 leader_election.cc:334] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.315s (SENT)
I20260806 03:43:59.225646 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.172020 4932 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.194169 4819 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.188207 4906 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260806 03:43:59.183030 3902 raft_consensus.cc:729] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:59.140211 4911 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260806 03:43:59.191728 766 raft_consensus.cc:2866] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:43:59.194324 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.233559 770 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:43:59.198736 546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "497d006930504e8184f1e5e512ade026"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:43:59.208837 661 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:59.234575 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1793) took 1193 ms. Trace:
I20260806 03:43:59.234619 546 raft_consensus.cc:595] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:43:59.240232 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:59.235102 668 rpcz_store.cc:276] 0806 03:43:58.041470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:58.067855 (+ 26385us) service_pool.cc:227] Handling call
0806 03:43:59.234330 (+1166475us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540015c2d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:59.234488 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":645676288}
I20260806 03:43:59.215162 498 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:59.185858 4871 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:43:59.224539 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 354679us
W20260806 03:43:59.199005 4636 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.616s user 0.011s sys 0.001s
W20260806 03:43:59.208449 4942 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260806 03:43:59.207558 4936 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.132s user 0.002s sys 0.000s
I20260806 03:43:59.196240 4863 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 3 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.207065 4927 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:59.235522 4761 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:43:59.196694 4757 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.245337 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:59.214828 4473 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.258718 4933 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:43:59.259521 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 70967.3us
I20260806 03:43:59.269093 4885 raft_consensus.cc:2741] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:43:59.250659 4903 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:59.269836 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 104032us
W20260806 03:43:59.231928 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 123554us
I20260806 03:43:59.209887 4772 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:43:59.297499 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 52241.6us
W20260806 03:43:59.299413 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 239490us
I20260806 03:43:59.300138 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:43:59.295965 4853 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.359s user 0.014s sys 0.020s
W20260806 03:43:59.324970 1622 leader_election.cc:334] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.825s (SENT)
I20260806 03:43:59.330559 1622 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:59.332056 1622 leader_election.cc:334] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.976s (SENT)
I20260806 03:43:59.332581 1622 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:59.324962 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:59.326694 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:59.329620 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:43:59.335023 1668 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:43:59.335606 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:59.265123 4945 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:43:59.339705 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:59.346616 674 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.341893 501 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:59.351377 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:59.354136 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.254321 4871 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.365805 1660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:43:59.361588 727 leader_election.cc:334] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.695s (SENT)
W20260806 03:43:59.337519 4853 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.400s user 0.014s sys 0.021s
I20260806 03:43:59.373129 669 raft_consensus.cc:1226] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 1. Current term is 3. Ops: []
I20260806 03:43:59.206876 4943 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.228134 4907 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.380846 727 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.347235 4927 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.358649 4658 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.347676 4473 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:59.392732 4929 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.393098 4473 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:59.393553 4473 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.394809 4929 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:43:59.396469 4935 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:59.396885 4871 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.397145 1665 raft_consensus.cc:1226] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 6. Current term is 7. Ops: []
I20260806 03:43:59.357345 4885 consensus_queue.cc:1046] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260806 03:43:59.399410 4885 raft_consensus.cc:3032] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:59.399771 4885 raft_consensus.cc:729] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.401937 4757 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.403074 4927 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:59.402505 400 catalog_manager.cc:5495] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:59.404433 400 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 4 to 6, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 6 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:59.387204 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 172141us
I20260806 03:43:59.344545 4945 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.384455 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:59.363384 4405 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:59.353797 4451 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:43:59.359632 4841 consensus_queue.cc:1046] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
W20260806 03:43:59.391563 4861 consensus_meta.cc:287] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.559s user 0.012s sys 0.000s
I20260806 03:43:59.413892 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.416378 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.419831 4861 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:59.420514 4951 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 03:43:59.410033 4963 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:59.423493 1658 raft_consensus.cc:1226] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:43:59.428630 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.426016 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.431248 767 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.430212 1622 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:43:59.434173 4949 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 03:43:59.301072 3902 consensus_queue.cc:260] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.414475 4945 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:43:59.436162 4871 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:59.467767 4871 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:59.468366 4871 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.474848 4907 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:59.476492 4931 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:43:59.443408 4933 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:43:59.477824 4933 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.478261 4933 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:43:59.415962 4497 consensus_queue.cc:1046] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260806 03:43:59.491267 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:59.491993 4497 raft_consensus.cc:3032] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:43:59.492532 4497 raft_consensus.cc:729] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.499985 4497 consensus_queue.cc:260] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.459575 4932 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:59.429586 4963 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:59.502007 4689 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.532611 4963 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.532946 4689 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.533565 4903 raft_consensus.cc:3032] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:59.534006 4903 raft_consensus.cc:729] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:43:59.534924 4903 consensus_queue.cc:260] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.536608 4903 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.528031 4906 consensus_queue.cc:1046] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260806 03:43:59.537930 4906 raft_consensus.cc:3032] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:43:59.460700 4861 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:43:59.538333 4906 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:43:59.461351 4959 consensus_queue.cc:1159] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "1914b652c9fb49b694d71a5f5f5752ee" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
I20260806 03:43:59.538559 4761 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.463057 4658 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.464587 4878 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.539148 4761 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.427052 4405 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.400554 4885 consensus_queue.cc:260] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:59.494014 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 80214.8us
I20260806 03:43:59.557852 498 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:59.503340 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 142221us
W20260806 03:43:59.559266 727 leader_election.cc:334] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 3.221s (SENT)
I20260806 03:43:59.559670 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:59.559824 1670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60272 (request call id 493) took 1514 ms (client timeout 1702 ms). Trace:
W20260806 03:43:59.560182 1670 rpcz_store.cc:269] 0806 03:43:58.045190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:58.045365 (+ 175us) service_pool.cc:227] Handling call
0806 03:43:59.559624 (+1514259us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b5400642654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:43:59.559765 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":952264064}
I20260806 03:43:59.504033 4922 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.457857 767 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:59.510047 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:59.508765 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 97922.2us
I20260806 03:43:59.569875 1657 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:43:59.528404 400 catalog_manager.cc:5495] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:59.526340 664 raft_consensus.cc:2454] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:43:59.540838 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 164324us
W20260806 03:43:59.494164 4268 consensus_meta.cc:287] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.622s user 0.013s sys 0.002s
I20260806 03:43:59.543521 674 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:43:59.574118 665 raft_consensus.cc:1261] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:43:59.557852 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1521) took 1064 ms. Trace:
I20260806 03:43:59.574864 766 rpcz_store.cc:276] 0806 03:43:58.493741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:58.521396 (+ 27655us) service_pool.cc:227] Handling call
0806 03:43:58.578973 (+ 57577us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:59.233087 (+654114us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:59.557594 (+324507us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54002bd4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:43:59.557778 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1391970048}
I20260806 03:43:59.539422 4906 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:59.584446 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 146434us
I20260806 03:43:59.582597 4510 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:43:59.618345 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46305.2us
I20260806 03:43:59.616300 400 catalog_manager.cc:5495] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 4 to 5, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 5 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
W20260806 03:43:59.627562 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 69805.3us
I20260806 03:43:59.630359 4934 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.634292 616 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:43:59.636098 4945 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.637735 4851 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:43:59.637632 4757 consensus_queue.cc:1035] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:43:59.638334 4878 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.638559 4975 raft_consensus.cc:2726] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.638325 4851 raft_consensus.cc:686] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:43:59.643607 614 leader_election.cc:334] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.646s (SENT)
I20260806 03:43:59.639804 4851 consensus_queue.cc:237] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.644196 614 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:59.644491 4473 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:43:59.644914 4853 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.645484 4853 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.651795 498 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:43:59.652966 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 93719.7us
W20260806 03:43:59.654695 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 283793us
I20260806 03:43:59.612327 4497 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.665719 772 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 03:43:59.655484 4945 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:59.650987 4757 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:43:59.663324 4431 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:43:59.664203 766 raft_consensus.cc:1261] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:43:59.661136 4885 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.733314 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.658785 4975 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:43:59.584637 4268 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:43:59.745036 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 84131.1us
I20260806 03:43:59.768100 501 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:43:59.748075 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 317942us
W20260806 03:43:59.752602 615 leader_election.cc:334] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823): Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.702s (SENT)
I20260806 03:43:59.769881 1622 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.770356 615 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:43:59.772122 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.739353 4473 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:43:59.780901 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.781662 768 raft_consensus.cc:3037] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.776230 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.781679 501 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:43:59.783601 4904 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:43:59.783444 4920 consensus_queue.cc:1046] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:43:59.738348 4853 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:43:59.786021 4932 raft_consensus.cc:2781] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:43:59.786603 4932 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.787005 4989 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:43:59.787042 4932 raft_consensus.cc:3037] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:43:59.787528 4989 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:59.787964 4989 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.791528 4978 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:43:59.775533 4931 consensus_queue.cc:1046] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260806 03:43:59.792994 4920 raft_consensus.cc:3032] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:59.793784 4920 raft_consensus.cc:729] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.794858 4920 consensus_queue.cc:260] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.796630 4920 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:43:59.800096 614 leader_election.cc:334] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.674s (SENT)
I20260806 03:43:59.800658 614 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:43:59.783047 772 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:43:59.807214 400 catalog_manager.cc:5495] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 1 to 4, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 4 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:43:59.812567 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:43:59.817353 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:43:59.818667 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:43:59.821206 661 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
I20260806 03:43:59.685463 4933 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.793265 4931 raft_consensus.cc:3032] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:43:59.831427 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:43:59.833238 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 100056us
I20260806 03:43:59.682734 4871 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:43:59.897161 768 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:59.911921 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.912629 776 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:43:59.913314 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.914304 768 raft_consensus.cc:2379] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:43:59.792263 4975 raft_consensus.cc:2781] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:43:59.797730 4658 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:43:59.892774 772 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 10.
I20260806 03:43:59.916718 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:59.918983 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:43:59.920439 1658 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 03:43:59.923137 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.923799 498 leader_election.cc:398] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9.
I20260806 03:43:59.937573 498 leader_election.cc:401] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260806 03:43:59.924182 771 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:43:59.937678 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.938582 768 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:43:59.939828 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 51259.2us
I20260806 03:43:59.936805 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:43:59.932125 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.942974 775 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:43:59.946797 1624 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:43:59.935838 1665 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:43:59.935851 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.953238 775 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:43:59.955835 768 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:59.927629 4975 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:43:59.957587 4975 raft_consensus.cc:3037] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:43:59.804581 4943 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:43:59.962703 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:43:59.827965 4927 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:43:59.997493 771 raft_consensus.cc:1226] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
W20260806 03:43:59.999547 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 107666us
I20260806 03:44:00.006225 661 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:43:59.776434 4431 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:00.009712 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 75823.5us
I20260806 03:43:59.915854 4931 raft_consensus.cc:729] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:43:59.811494 4978 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.168s
I20260806 03:43:59.927441 4981 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.018361 1657 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:43:59.972388 4689 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.050117 663 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:43:59.797746 4405 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:00.026978 4936 log.cc:924] Time spent T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260806 03:44:00.013514 4761 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.096561 663 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:00.103142 4965 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:00.103714 4933 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:00.106515 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 452294us
I20260806 03:44:00.023676 4473 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:43:59.997892 4943 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.011004 4927 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.097870 615 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:44:00.111143 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 102706us
I20260806 03:43:59.981585 4934 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
I20260806 03:44:00.133030 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.135027 661 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Advancing to term 11
W20260806 03:44:00.124795 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 53175.8us
I20260806 03:44:00.143318 400 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e reported cstate change: term changed from 2 to 3, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:00.096138 5001 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:00.102186 4772 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:00.100708 4945 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260806 03:44:00.134320 4788 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:00.156378 614 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.167737 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:00.185884 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:00.149804 5001 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.151382 4928 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:00.115953 4926 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260806 03:44:00.186596 1671 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:44:00.192353 4992 log.cc:924] Time spent T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260806 03:44:00.122478 4473 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:00.207544 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 289294us
I20260806 03:44:00.185299 4772 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.182514 4788 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.222432 4788 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.187600 4934 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.182682 4978 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9.
W20260806 03:44:00.206733 4497 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.546s user 0.010s sys 0.000s
I20260806 03:44:00.200202 4928 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:00.199558 5001 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:00.206941 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 232328us
W20260806 03:44:00.236086 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 51595.8us
I20260806 03:44:00.249008 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.252382 498 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.216969 4871 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:00.257472 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 47706.1us
I20260806 03:44:00.272078 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:00.272913 772 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.276582 776 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
I20260806 03:44:00.279405 5013 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 03:44:00.279841 4982 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.357s user 0.001s sys 0.000s
I20260806 03:44:00.280381 4473 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.280784 4861 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:44:00.281246 4861 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:00.283097 498 leader_election.cc:398] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5.
I20260806 03:44:00.283465 498 leader_election.cc:401] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:44:00.217197 4853 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
W20260806 03:44:00.274852 4963 consensus_meta.cc:287] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.741s user 0.014s sys 0.000s
I20260806 03:44:00.301052 4871 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:00.272881 4268 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.266548 4497 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:44:00.292860 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 109028us
W20260806 03:44:00.308214 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 59342.6us
I20260806 03:44:00.322772 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 419) took 1587 ms. Trace:
W20260806 03:44:00.321821 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 48594us
I20260806 03:44:00.281698 4978 consensus_queue.cc:1046] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260806 03:44:00.322281 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.324563 547 rpcz_store.cc:276] 0806 03:43:58.735859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:58.736077 (+ 218us) service_pool.cc:227] Handling call
0806 03:43:58.736561 (+ 484us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:59.538968 (+802407us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b1800143b60. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f3215fae82c 00007f32133d99dc 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f32163173d0 00007f3216317f6b 00007f321632278b
0806 03:43:59.539138 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:59.539167 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:59.539796 (+ 629us) log.cc:842] Serialized 52 byte log entry
0806 03:43:59.888315 (+348519us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:00.026973 (+138658us) log.cc:842] Serialized 14 byte log entry
0806 03:44:00.134692 (+107719us) log.cc:842] Serialized 14 byte log entry
0806 03:44:00.147797 (+ 13105us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:00.147882 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:00.322537 (+174655us) raft_consensus.cc:1671] finished
0806 03:44:00.322564 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:00.322712 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85329,"prepare.queue_time_us":225117,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":88,"raft.queue_time_us":375885,"raft.run_cpu_time_us":443,"raft.run_wall_time_us":26782,"spinlock_wait_cycles":11337472,"thread_start_us":655498,"threads_started":4,"wal-append.queue_time_us":456930}
I20260806 03:44:00.307034 4431 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.149490 4931 consensus_queue.cc:260] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.239894 4934 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.361248 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:00.354993 776 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:00.362090 1657 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:00.318872 4927 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.360775 661 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 11.
I20260806 03:44:00.312927 4929 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5.
I20260806 03:44:00.406523 772 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:00.304862 4963 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.366592 4786 consensus_queue.cc:1035] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:44:00.367349 4986 log.cc:924] Time spent T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.249s user 0.000s sys 0.001s
I20260806 03:44:00.367722 4944 raft_consensus.cc:2726] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:44:00.368345 4985 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.289s user 0.001s sys 0.000s
I20260806 03:44:00.367822 5025 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
W20260806 03:44:00.374346 4987 log.cc:924] Time spent T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.280s user 0.001s sys 0.001s
I20260806 03:44:00.455626 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 498) took 1143 ms. Trace:
I20260806 03:44:00.456516 4934 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.456209 1662 rpcz_store.cc:276] 0806 03:43:59.312595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:59.323811 (+ 11216us) service_pool.cc:227] Handling call
0806 03:43:59.383116 (+ 59305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:59.420380 (+ 37264us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:59.420417 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:59.420928 (+ 511us) log.cc:842] Serialized 26 byte log entry
0806 03:43:59.663560 (+242632us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:59.690580 (+ 27020us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:59.737397 (+ 46817us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:00.455373 (+717976us) raft_consensus.cc:1671] finished
0806 03:44:00.455404 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:00.455559 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7961,"raft.queue_time_us":11401,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":16294,"thread_start_us":235310,"threads_started":1,"wal-append.queue_time_us":512333}
I20260806 03:44:00.327509 498 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.392105 1624 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:00.459152 4934 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:44:00.459312 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 131884us
I20260806 03:44:00.459791 4934 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.460184 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.456717 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1548) took 1081 ms. Trace:
I20260806 03:44:00.460428 4934 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:00.460788 770 rpcz_store.cc:276] 0806 03:43:59.375795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:59.376777 (+ 982us) service_pool.cc:227] Handling call
0806 03:43:59.406408 (+ 29631us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:43:59.504732 (+ 98324us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:59.504761 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:43:59.505221 (+ 460us) log.cc:842] Serialized 26 byte log entry
0806 03:43:59.737300 (+232079us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:43:59.737341 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:59.737420 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:00.456513 (+719093us) raft_consensus.cc:1671] finished
0806 03:44:00.456532 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:00.456660 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118943,"raft.queue_time_us":98641,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":231724,"threads_started":1,"wal-append.queue_time_us":320403}
W20260806 03:44:00.394026 4996 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.195s user 0.000s sys 0.000s
I20260806 03:44:00.463706 4451 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:44:00.464087 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 72154.3us
I20260806 03:44:00.464664 498 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.470965 4861 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.471935 498 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.472301 4861 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:00.473055 498 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.473613 766 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260806 03:44:00.474149 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:00.474761 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.468149 728 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:00.468461 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.476895 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.468283 4922 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.478363 4922 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.480448 4922 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:00.480953 4922 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:00.481287 4922 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.481592 4922 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:00.469017 4909 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.483083 4909 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:00.484306 4903 consensus_meta.cc:287] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.947s user 0.008s sys 0.005s
I20260806 03:44:00.484879 669 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:00.464368 4451 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:00.486801 4689 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:00.487188 4689 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:00.487553 4885 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.707s user 0.010s sys 0.000s
I20260806 03:44:00.487816 4761 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:44:00.439724 4931 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:44:00.489117 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.489768 770 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:00.487568 4689 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:00.491149 4929 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:00.491743 4929 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.492314 4929 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:00.493750 498 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.496248 4451 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:44:00.496394 4989 consensus_meta.cc:287] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.708s user 0.007s sys 0.004s
W20260806 03:44:00.497203 1622 leader_election.cc:334] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.891s (SENT)
I20260806 03:44:00.497568 498 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:00.498436 5021 raft_consensus.cc:3032] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:44:00.498682 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36554 (request call id 1799) took 1199 ms. Trace:
I20260806 03:44:00.498867 5021 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:00.499759 5021 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.485109 4909 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:00.502800 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2424) took 1392 ms. Trace:
I20260806 03:44:00.503216 665 rpcz_store.cc:276] 0806 03:43:59.110216 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:59.171213 (+ 60997us) service_pool.cc:227] Handling call
0806 03:43:59.572148 (+400935us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:43:59.574542 (+ 2394us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:00.502604 (+928062us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b540016e4d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:00.502743 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":878604672}
I20260806 03:44:00.503883 4978 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:00.504282 4978 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.504626 4978 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:00.504959 4906 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:00.505477 4451 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:00.505913 4975 consensus_meta.cc:287] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.547s user 0.007s sys 0.000s
I20260806 03:44:00.505982 4451 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.507010 4983 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.507503 4983 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.508368 4451 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.509582 4983 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:00.459084 4932 consensus_meta.cc:287] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.671s user 0.012s sys 0.001s
I20260806 03:44:00.512861 4929 consensus_queue.cc:1035] T 20fcbc41710a42cd9ee5dacc5d9577fb P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:44:00.513422 4932 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.515050 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.515455 4932 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.515740 542 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:00.499079 668 rpcz_store.cc:276] 0806 03:43:59.299404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:59.299570 (+ 166us) service_pool.cc:227] Handling call
0806 03:43:59.328511 (+ 28941us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:43:59.374423 (+ 45912us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:43:59.374454 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:43:59.381358 (+ 6904us) log.cc:842] Serialized 52 byte log entry
0806 03:43:59.661437 (+280079us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:43:59.711240 (+ 49803us) log.cc:842] Serialized 14 byte log entry
0806 03:43:59.733025 (+ 21785us) log.cc:842] Serialized 14 byte log entry
0806 03:43:59.738048 (+ 5023us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:43:59.738126 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:00.427492 (+689366us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b18004ccc00. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:44:00.462292 (+ 34800us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18004ccc00. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133c4f96 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11
0806 03:44:00.462745 (+ 453us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b18004ccc00. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:44:00.498473 (+ 35728us) raft_consensus.cc:1671] finished
0806 03:44:00.498495 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:00.498625 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99291,"prepare.queue_time_us":85538,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"raft.queue_time_us":70587,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":41813,"spinlock_wait_cycles":115221760,"thread_start_us":251119,"threads_started":1,"wal-append.queue_time_us":374394}
I20260806 03:44:00.515996 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.516839 546 raft_consensus.cc:2396] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:00.516446 4788 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.517560 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.518170 558 raft_consensus.cc:2362] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6. Re-sending same reply.
I20260806 03:44:00.518431 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.519207 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.520212 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.520784 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.521149 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.521596 558 raft_consensus.cc:2454] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:00.521837 563 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
W20260806 03:44:00.417708 4993 log.cc:924] Time spent T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.272s user 0.003s sys 0.005s
I20260806 03:44:00.404772 4943 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.522770 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.523593 542 raft_consensus.cc:2379] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:44:00.523824 4658 consensus_meta.cc:287] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.554s user 0.010s sys 0.001s
I20260806 03:44:00.435739 4963 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.378791 4772 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.524442 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.524905 4658 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:44:00.524981 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.525599 1663 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:44:00.526335 561 raft_consensus.cc:1261] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:44:00.310751 4928 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.527797 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.521942 546 raft_consensus.cc:1261] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260806 03:44:00.528654 561 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.424750 3902 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:00.537614 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.540992 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.541744 547 raft_consensus.cc:2362] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9. Re-sending same reply.
W20260806 03:44:00.543172 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 487020us
I20260806 03:44:00.544175 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.541819 563 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:00.542243 564 raft_consensus.cc:1261] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:44:00.543068 548 raft_consensus.cc:2396] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260806 03:44:00.363854 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 46224.4us
I20260806 03:44:00.555682 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.557225 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:00.557855 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.558251 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.557818 1665 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.561172 558 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.557256 564 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:44:00.562672 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 245083us
I20260806 03:44:00.538880 5021 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:44:00.563344 616 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.564317 1624 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:00.554919 499 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:00.566606 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.567658 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:00.556473 772 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:44:00.569360 766 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:44:00.569429 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.569355 501 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.571400 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.573415 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.574664 614 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.520243 4788 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.575459 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.579790 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.572073 564 raft_consensus.cc:2454] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:00.582034 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.572067 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:00.582792 564 raft_consensus.cc:2354] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:00.583437 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.567754 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:44:00.554579 4832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.008s sys 0.006s
I20260806 03:44:00.585171 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:00.587045 771 raft_consensus.cc:2379] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:00.587178 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.586542 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.587246 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.596256 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.596918 564 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:00.597787 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.497246 4989 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.547127 4975 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.558472 5001 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.644084 1665 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:00.612498 4772 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:44:00.600917 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:00.661789 1670 raft_consensus.cc:2379] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:00.603111 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.598801 563 raft_consensus.cc:2379] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:44:00.631887 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 59893.9us
I20260806 03:44:00.650035 564 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:00.630756 548 raft_consensus.cc:2354] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:00.665164 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.668831 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.583575 4945 raft_consensus.cc:2781] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:00.586731 4963 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.672194 615 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.672325 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:00.672830 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 75294.8us
I20260806 03:44:00.671789 667 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:44:00.672535 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.681370 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.681578 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:00.583284 4861 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:00.608281 4928 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.689558 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.690894 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.671780 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.588651 4931 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:00.582036 4841 consensus_queue.cc:1035] T e8b626bb795440e18f02bbdcbd10f9ce P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20260806 03:44:00.604120 5026 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:00.613991 4920 consensus_meta.cc:287] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.817s user 0.012s sys 0.001s
I20260806 03:44:00.680302 1657 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:00.597707 4932 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.683270 562 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.585099 4904 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.685029 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.685516 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.685756 1671 raft_consensus.cc:1226] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:00.599726 5035 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:00.612185 4981 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:00.592067 4832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.621s user 0.009s sys 0.006s
I20260806 03:44:00.622098 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.697360 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.698151 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.740393 548 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:44:00.673372 4772 raft_consensus.cc:686] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:00.699991 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.614388 4863 consensus_queue.cc:1035] T fb67a142feaf496f862fbd7f06a925f9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260806 03:44:00.700793 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.701709 668 raft_consensus.cc:1261] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:44:00.684899 4953 raft_consensus.cc:2781] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:00.597501 4473 consensus_queue.cc:1046] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260806 03:44:00.604074 4929 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260806 03:44:00.763543 4975 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.603834 4788 consensus_queue.cc:1035] T 93b5917270cd4146bb9b6c0ac0dc8414 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20260806 03:44:00.686542 4861 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.687767 4405 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:00.595252 4911 consensus_queue.cc:1035] T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.108s
I20260806 03:44:00.596856 4907 consensus_queue.cc:1035] T 2c5936af1dcf436eb5f06e8fc0765e83 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260806 03:44:00.685256 4945 raft_consensus.cc:686] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:00.724051 544 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:44:00.725422 564 raft_consensus.cc:2379] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.782680 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.778270 563 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:00.676424 4988 consensus_queue.cc:1035] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260806 03:44:00.676277 4658 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260806 03:44:00.679411 4926 consensus_queue.cc:1035] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260806 03:44:00.677619 4935 consensus_queue.cc:1035] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260806 03:44:00.737027 669 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:00.704464 498 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.606288 4978 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.709229 3902 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:00.746484 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:00.746510 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 269539us
I20260806 03:44:00.749979 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.752735 4772 consensus_queue.cc:237] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.820776 4945 consensus_queue.cc:237] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.823315 4922 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.827034 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:00.827409 4978 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.771281 5005 consensus_queue.cc:1046] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.058s
I20260806 03:44:00.829834 5005 raft_consensus.cc:3032] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:00.830274 5005 raft_consensus.cc:729] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:00.831302 5005 consensus_queue.cc:260] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.832783 5005 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:00.831493 4922 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:00.839182 4989 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:00.754920 4946 consensus_queue.cc:1035] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260806 03:44:00.757593 4841 consensus_queue.cc:1046] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:00.758183 4953 raft_consensus.cc:686] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:00.764036 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:00.715441 4931 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:00.771646 1622 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.775705 766 raft_consensus.cc:1261] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 03:44:00.855877 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 84318.7us
I20260806 03:44:00.722486 5011 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.709915 4963 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.040s
I20260806 03:44:00.777859 548 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 11.
I20260806 03:44:00.863529 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.863924 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:00.864228 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.779572 772 raft_consensus.cc:1261] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:44:00.790283 4905 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:00.785684 564 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:00.772604 4861 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:00.725805 4934 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:00.802912 562 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:00.803030 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.787830 4689 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.804404 4451 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:44:00.765100 542 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:00.716147 5026 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:00.809583 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.891424 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:44:00.815066 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 110901us
I20260806 03:44:00.814985 5001 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:00.816407 4975 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:00.845111 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.898094 668 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.847951 614 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.854955 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.857184 1622 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.899498 544 raft_consensus.cc:3037] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:44:00.899751 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 51897.1us
I20260806 03:44:00.899890 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:00.859277 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.860935 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:00.780452 773 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:44:00.901479 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:00.867215 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:00.867764 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.902719 1670 raft_consensus.cc:2379] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:00.868005 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:00.742411 4981 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:00.887984 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 85049.2us
I20260806 03:44:00.888427 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.892714 498 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.815407 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:00.918802 775 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 3: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:00.919067 498 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.894243 3902 raft_consensus.cc:3032] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:00.903152 548 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.865796 4832 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Leader election won for term 11
I20260806 03:44:00.915028 766 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:00.917850 564 raft_consensus.cc:3037] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:00.894619 558 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:00.894433 4922 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:00.938338 498 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.929075 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.939870 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.941017 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.931005 727 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:00.944725 727 leader_election.cc:398] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 8. Current term is 9.
W20260806 03:44:00.952579 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 51161.1us
I20260806 03:44:00.937791 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:00.953528 614 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:00.903111 4451 consensus_queue.cc:1225] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:44:00.955797 667 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:00.934041 562 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:00.945132 727 leader_election.cc:401] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:44:00.959527 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:00.959843 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:01.022905 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 68689.2us
I20260806 03:44:00.893438 4905 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.022675 564 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:00.952046 4922 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.976946 1657 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:01.002807 544 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:00.930413 5001 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:01.033078 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.936679 5009 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:00.934352 5044 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:00.867481 5011 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.035679 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.928107 3902 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:01.035679 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:01.029963 561 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:44:00.952409 4832 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
W20260806 03:44:01.040678 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 96137us
I20260806 03:44:01.028745 5035 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:00.978286 4920 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:00.936622 4975 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:00.955403 5020 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:44:01.044481 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 144668us
I20260806 03:44:00.965701 4757 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.033033 5051 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 8. Current term is 9.
I20260806 03:44:00.963762 4932 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.043823 562 raft_consensus.cc:2362] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8. Re-sending same reply.
I20260806 03:44:00.972779 4405 consensus_queue.cc:1035] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.084s
I20260806 03:44:00.974160 4903 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.022507 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:00.973903 4935 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:01.033684 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:01.061923 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.078418 659 raft_consensus.cc:3037] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:01.043017 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:01.079329 544 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:01.079610 564 raft_consensus.cc:3032] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:44:01.081941 564 raft_consensus.cc:729] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:00.957259 4953 consensus_queue.cc:237] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.077461 562 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:44:01.050763 5009 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:01.082994 562 raft_consensus.cc:595] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:01.083418 562 raft_consensus.cc:603] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:44:01.039296 4819 consensus_queue.cc:1035] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260806 03:44:01.088701 544 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:01.065845 1622 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:01.059589 5035 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:00.965299 4931 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.073213 498 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:01.100090 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:01.043443 4786 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260806 03:44:01.055904 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:01.055044 5001 raft_consensus.cc:686] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:01.100852 558 raft_consensus.cc:2362] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5. Re-sending same reply.
I20260806 03:44:01.111662 674 raft_consensus.cc:1261] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:44:01.113933 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:44:01.110148 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 54329.7us
I20260806 03:44:01.123618 727 leader_election.cc:398] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:44:01.058820 4119 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260806 03:44:01.132929 776 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:44:01.127645 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.135535 773 raft_consensus.cc:2379] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:01.124092 727 leader_election.cc:401] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:44:01.084710 5009 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:01.131980 768 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:01.139546 562 raft_consensus.cc:1226] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:01.083453 4935 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.108265 5035 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:01.147930 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:01.124366 542 raft_consensus.cc:2379] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 3: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:01.099634 5048 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.168372 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.153821 773 raft_consensus.cc:2379] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:01.157805 5026 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:01.108019 5020 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:01.131011 4903 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.057229 5044 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.163296 659 raft_consensus.cc:2454] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:01.087600 5032 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260806 03:44:01.128281 5001 consensus_queue.cc:237] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.145753 4832 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.140990 4689 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:01.141568 5052 log.cc:924] Time spent T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
W20260806 03:44:01.143170 5059 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260806 03:44:01.143745 5054 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.189s user 0.000s sys 0.000s
W20260806 03:44:01.144446 5055 log.cc:924] Time spent T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20260806 03:44:01.176776 772 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:01.161862 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.186769 498 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:01.228590 558 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:44:01.229234 558 raft_consensus.cc:595] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:01.229578 558 raft_consensus.cc:603] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:44:01.187526 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.230103 773 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:01.147782 4473 consensus_queue.cc:1035] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.071s
I20260806 03:44:01.188354 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:01.141319 4963 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20260806 03:44:01.235080 616 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:01.237108 616 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:01.181733 4829 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) with higher term. Message: Invalid argument: T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 2. Current term is 3.
I20260806 03:44:01.181388 4863 consensus_queue.cc:1046] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260806 03:44:01.165230 4920 consensus_queue.cc:1046] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260806 03:44:01.190351 544 raft_consensus.cc:1261] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:44:01.152130 4934 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:01.152971 4909 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:01.217679 4981 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:01.220011 4935 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:01.131327 3902 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.232968 5021 raft_consensus.cc:2781] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:01.238447 4963 consensus_queue.cc:1225] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:44:01.249816 4119 raft_consensus.cc:3032] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:44:01.307998 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 145008us
W20260806 03:44:01.255498 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 77988.3us
I20260806 03:44:01.260107 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.142323 5070 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:01.248211 668 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
W20260806 03:44:01.268005 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 202239us
I20260806 03:44:01.238732 4841 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:01.312335 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:01.316671 1665 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:01.311952 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:01.257483 4819 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:01.251884 4932 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:44:01.318395 615 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:01.322933 558 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:44:01.323547 558 raft_consensus.cc:595] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:01.323851 558 raft_consensus.cc:603] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:44:01.312127 727 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:01.319823 1622 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:01.181479 4975 consensus_queue.cc:1046] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260806 03:44:01.227846 4861 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.257244 4905 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:01.192390 5020 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.328092 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.337832 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:01.319701 5059 log.cc:924] Time spent T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 03:44:01.338438 5021 raft_consensus.cc:686] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:01.322252 4963 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.266815 4909 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:01.419445 772 raft_consensus.cc:2379] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 4: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:01.639797 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1396) took 1099 ms. Trace:
I20260806 03:44:01.688747 544 rpcz_store.cc:276] 0806 03:44:00.540104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.097822 (+557718us) service_pool.cc:227] Handling call
0806 03:44:01.166158 (+ 68336us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:01.243253 (+ 77095us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.639465 (+396212us) spinlock_profiling.cc:243] Waited 67.9 ms on lock 0x7b54002b1950. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:01.639705 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156150528}
I20260806 03:44:01.618929 659 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: [5.2-5.2]
W20260806 03:44:01.604574 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 292529us
I20260806 03:44:01.627933 563 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:44:01.620229 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1389) took 1090 ms. Trace:
I20260806 03:44:01.447609 1662 raft_consensus.cc:1226] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
W20260806 03:44:01.587363 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 74177.8us
I20260806 03:44:01.587040 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:01.420650 1665 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260806 03:44:01.693944 561 rpcz_store.cc:276] 0806 03:44:00.529613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.538501 (+ 8888us) service_pool.cc:227] Handling call
0806 03:44:01.620007 (+1081506us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54000cf154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:01.620152 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":917220608}
I20260806 03:44:01.695504 1665 pending_rounds.cc:76] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting all ops after (but not including) 7
I20260806 03:44:01.696753 1665 pending_rounds.cc:97] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Aborting uncommitted NO_OP operation due to leader change: 9.8
W20260806 03:44:01.693966 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 374198us
I20260806 03:44:01.699455 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:01.698156 1665 raft_consensus.cc:2866] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:44:01.309856 4119 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:01.704357 561 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:01.707965 1622 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:01.708840 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.718361 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:01.707782 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2368) took 1143 ms. Trace:
I20260806 03:44:01.719797 770 rpcz_store.cc:276] 0806 03:44:00.564677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.564800 (+ 123us) service_pool.cc:227] Handling call
0806 03:44:00.565139 (+ 339us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:44:00.565537 (+ 398us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:00.565566 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:44:00.566140 (+ 574us) log.cc:842] Serialized 52 byte log entry
0806 03:44:00.850621 (+284481us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:00.850668 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:00.850744 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:01.476424 (+625680us) raft_consensus.cc:1671] finished
0806 03:44:01.476452 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:01.707717 (+231265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":260,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":22334,"thread_start_us":235056,"threads_started":1,"wal-append.queue_time_us":349458}
I20260806 03:44:01.322384 4819 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:01.326839 4851 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:01.714999 727 connection.cc:232] RPC call timeout handler was delayed by -2.344096670858562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
I20260806 03:44:01.726022 1622 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:01.733893 727 leader_election.cc:334] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.506s (SENT)
W20260806 03:44:01.735021 727 leader_election.cc:334] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.585s (SENT)
I20260806 03:44:01.735723 727 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:01.353084 3902 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:01.419637 5052 log.cc:924] Time spent T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260806 03:44:01.718349 558 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 4.
I20260806 03:44:01.342669 5011 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:01.616736 5044 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:01.455981 4829 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:01.742414 767 raft_consensus.cc:1261] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 03:44:01.363730 5070 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.655846 4922 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:01.634701 4761 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:01.750730 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.457140 5080 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:01.499089 4689 raft_consensus.cc:2741] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 03:44:01.761416 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:01.774806 727 connection.cc:232] RPC call timeout handler was delayed by -2.21809667092748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
I20260806 03:44:01.776075 544 raft_consensus.cc:2396] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:01.776233 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.777175 558 raft_consensus.cc:2379] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:01.728565 4935 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:01.581722 4861 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:01.768513 561 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:44:01.739943 4819 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:44:01.776446 727 connection.cc:232] RPC call timeout handler was delayed by -2.1901966708246619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
W20260806 03:44:01.793251 727 connection.cc:232] RPC call timeout handler was delayed by -2.1325966708827764s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260806 03:44:01.789454 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1390) took 1259 ms. Trace:
I20260806 03:44:01.794673 546 rpcz_store.cc:276] 0806 03:44:00.529913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.534583 (+ 4670us) service_pool.cc:227] Handling call
0806 03:44:00.534947 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:00.535428 (+ 481us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:00.535455 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:00.535908 (+ 453us) log.cc:842] Serialized 26 byte log entry
0806 03:44:00.776780 (+240872us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:00.792676 (+ 15896us) log.cc:842] Serialized 14 byte log entry
0806 03:44:00.835376 (+ 42700us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:00.838623 (+ 3247us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:01.789207 (+950584us) raft_consensus.cc:1671] finished
0806 03:44:01.789231 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:01.789387 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40198,"prepare.queue_time_us":38560,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":45795,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":11420,"thread_start_us":240479,"threads_started":1,"wal-append.queue_time_us":359892}
W20260806 03:44:01.776860 5073 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.452s user 0.001s sys 0.000s
I20260806 03:44:01.750841 4832 raft_consensus.cc:2781] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:01.787866 614 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:01.718827 4863 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260806 03:44:01.760512 4934 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:01.766479 770 raft_consensus.cc:1261] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:44:01.607311 5021 consensus_queue.cc:237] T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:01.761111 4953 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:01.807061 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:01.812042 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 70900.2us
I20260806 03:44:01.770444 5011 consensus_queue.cc:1046] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260806 03:44:01.777244 4405 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.733076 4963 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.813042 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.812944 499 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:01.818193 727 connection.cc:232] RPC call timeout handler was delayed by -2.0119966708589345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20260806 03:44:01.795243 5080 raft_consensus.cc:686] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:01.785719 4658 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:01.803115 4861 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.805166 4757 consensus_queue.cc:1035] T e12cd5292fc847be9a0e9d515b7faf96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
W20260806 03:44:01.851545 727 connection.cc:232] RPC call timeout handler was delayed by -1.9561966708861291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
W20260806 03:44:01.952672 727 connection.cc:232] RPC call timeout handler was delayed by -1.931896670954302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20260806 03:44:01.953006 727 connection.cc:232] RPC call timeout handler was delayed by -1.8598966710269451s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.207s for a response
W20260806 03:44:01.953299 727 connection.cc:232] RPC call timeout handler was delayed by -1.8490966709796339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260806 03:44:01.953583 727 connection.cc:232] RPC call timeout handler was delayed by -1.7185966710094362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20260806 03:44:01.819553 5051 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:44:01.905254 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 92406.3us
I20260806 03:44:01.775331 4761 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:01.775135 4922 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:01.843196 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1412) took 1057 ms. Trace:
I20260806 03:44:01.948076 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:01.949391 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 104183us
I20260806 03:44:01.964844 1622 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:01.957358 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.842612 4832 raft_consensus.cc:686] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:01.959192 771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1570) took 1190 ms. Trace:
I20260806 03:44:01.966313 771 rpcz_store.cc:276] 0806 03:44:00.768539 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.768764 (+ 225us) service_pool.cc:227] Handling call
0806 03:44:00.769059 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:00.769416 (+ 357us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:00.769442 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:00.780815 (+ 11373us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.180805 (+399990us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:44:01.180849 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.180922 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:01.958906 (+777984us) raft_consensus.cc:1671] finished
0806 03:44:01.958936 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:01.959116 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":128,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":6613,"thread_start_us":329659,"threads_started":1,"wal-append.queue_time_us":457806}
I20260806 03:44:01.954452 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:01.861863 5011 consensus_queue.cc:1225] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:44:01.963749 546 rpcz_store.cc:276] 0806 03:44:00.786194 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.806420 (+1020226us) service_pool.cc:227] Handling call
0806 03:44:01.842701 (+ 36281us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:01.842884 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.842914 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:01.842955 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:01.842990 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.843050 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:01.843141 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:01.771641 5070 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.976521 1622 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:01.778928 4119 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:01.947576 5026 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:44:01.986857 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:44:01.958613 4928 consensus_meta.cc:287] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.258s user 0.008s sys 0.006s
W20260806 03:44:01.952416 5082 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.443s user 0.003s sys 0.006s
W20260806 03:44:01.984728 5103 log.cc:924] Time spent T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
W20260806 03:44:01.984817 5084 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.286s user 0.001s sys 0.000s
I20260806 03:44:01.997723 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:01.849445 4953 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.999339 558 raft_consensus.cc:1261] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:44:01.999449 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:01.964313 4473 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:01.983242 5005 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.150s user 0.017s sys 0.001s
I20260806 03:44:02.006307 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 527) took 1031 ms. Trace:
I20260806 03:44:01.957053 4861 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:02.011945 542 raft_consensus.cc:1226] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:01.862327 4829 raft_consensus.cc:686] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:02.013159 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1416) took 1200 ms. Trace:
I20260806 03:44:01.993108 5100 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:44:02.014847 730 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.028s
W20260806 03:44:02.038460 730 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.588s user 0.000s sys 0.028s
I20260806 03:44:01.975699 4451 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.895556 4405 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:01.952042 5117 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:02.012830 5105 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:01.978288 4988 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 03:44:01.993302 5075 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.197s user 0.000s sys 0.001s
I20260806 03:44:02.025712 546 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:01.963819 4658 raft_consensus.cc:686] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:02.022244 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1414) took 1210 ms. Trace:
I20260806 03:44:02.088430 558 rpcz_store.cc:276] 0806 03:44:00.812000 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.987753 (+1175753us) service_pool.cc:227] Handling call
0806 03:44:01.999024 (+ 11271us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.021973 (+ 22949us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:02.022187 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:02.024914 661 rpcz_store.cc:276] 0806 03:44:00.956881 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.026878 (+ 69997us) service_pool.cc:227] Handling call
0806 03:44:01.982287 (+955409us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54002ae254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4af0 00007f3216327f2a 00007f321b264c6e 00007f321495b807 00007f321495b792 00007f321495b6c1 00007f321495b63c 00007f321495a8f2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:01.988114 (+ 5827us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":671458176}
I20260806 03:44:01.967756 5034 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:02.033533 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1409) took 1322 ms. Trace:
I20260806 03:44:02.092809 542 rpcz_store.cc:276] 0806 03:44:00.710769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.478179 (+767410us) service_pool.cc:227] Handling call
0806 03:44:02.004843 (+526664us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.033276 (+ 28433us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:02.033475 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:02.029325 5020 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:02.034082 548 raft_consensus.cc:2396] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:44:02.033080 544 rpcz_store.cc:276] 0806 03:44:00.812612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.012084 (+1199472us) service_pool.cc:227] Handling call
0806 03:44:02.012524 (+ 440us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.012759 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:02.012798 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:02.012854 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:02.012901 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:02.012968 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:02.013090 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:02.007192 5043 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:02.009801 4119 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:02.084537 4829 consensus_queue.cc:237] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:02.085801 5074 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:02.020138 5094 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:02.033257 4909 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:02.099340 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1413) took 1287 ms. Trace:
W20260806 03:44:02.100766 5092 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
I20260806 03:44:02.061014 4451 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:02.038167 5100 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:02.041260 5080 consensus_queue.cc:237] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:02.175707 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 94439.4us
I20260806 03:44:02.104420 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1419) took 1282 ms. Trace:
W20260806 03:44:02.328176 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 309553us
I20260806 03:44:02.968549 544 rpcz_store.cc:276] 0806 03:44:00.813491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.095013 (+1281522us) service_pool.cc:227] Handling call
0806 03:44:02.095402 (+ 389us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.095622 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:02.095653 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:02.095699 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:02.095752 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:02.095820 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:02.095957 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:02.688562 668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.007s sys 0.013s
W20260806 03:44:02.970175 668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.950s user 0.008s sys 0.019s
I20260806 03:44:02.607648 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2377) took 1584 ms. Trace:
I20260806 03:44:02.958163 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2442) took 1781 ms. Trace:
I20260806 03:44:02.971287 546 rpcz_store.cc:276] 0806 03:44:00.811613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.979793 (+1168180us) service_pool.cc:227] Handling call
0806 03:44:02.020875 (+ 41082us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.099056 (+ 78181us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:02.099269 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:02.837977 1670 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 03:44:02.335309 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 380944us
I20260806 03:44:02.971333 674 rpcz_store.cc:276] 0806 03:44:01.176506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.176711 (+ 205us) service_pool.cc:227] Handling call
0806 03:44:01.215959 (+ 39248us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:44:01.216523 (+ 564us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.216563 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:44:01.227435 (+ 10872us) log.cc:842] Serialized 52 byte log entry
0806 03:44:01.351076 (+123641us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:01.351223 (+ 147us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.361631 (+ 10408us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:02.957876 (+1596245us) raft_consensus.cc:1671] finished
0806 03:44:02.957900 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:02.958092 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1614,"raft.queue_time_us":145,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":10793,"thread_start_us":84736,"threads_started":1,"wal-append.queue_time_us":339142}
I20260806 03:44:02.974864 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:02.459136 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s user 0.002s sys 0.011s
I20260806 03:44:02.975556 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:44:02.975991 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.173s user 0.003s sys 0.024s
W20260806 03:44:02.976289 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 1.17355e+06us
W20260806 03:44:02.726292 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.693s user 0.017s sys 0.003s
W20260806 03:44:02.977120 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.943s user 0.019s sys 0.006s
I20260806 03:44:02.981199 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:02.684154 618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.005s sys 0.031s
I20260806 03:44:02.992870 770 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:44:02.992899 618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.049s user 0.006s sys 0.036s
I20260806 03:44:03.002620 727 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:02.971041 775 rpcz_store.cc:276] 0806 03:44:01.023232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.023464 (+ 232us) service_pool.cc:227] Handling call
0806 03:44:01.023827 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.029581 (+ 5754us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:44:01.171929 (+142348us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b18002a0340. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f4155 00007f3219cf4167 00007f3216329807 00007f3216338f0f 00007f3216338e88
0806 03:44:01.315291 (+143362us) log.cc:842] Serialized 14 byte log entry
0806 03:44:01.810639 (+495348us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.814432 (+ 3793us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.814862 (+ 430us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:01.814945 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.815130 (+ 185us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:02.607206 (+792076us) raft_consensus.cc:1671] finished
0806 03:44:02.607372 (+ 166us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:02.607566 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35985,"prepare.queue_time_us":113498,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":286799,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":29085,"spinlock_wait_cycles":11617152,"thread_start_us":532313,"threads_started":3,"wal-append.queue_time_us":1135227}
W20260806 03:44:02.437224 5035 consensus_meta.cc:287] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.265s user 0.012s sys 0.010s
W20260806 03:44:02.591166 5128 log.cc:924] Time spent T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.271s user 0.000s sys 0.000s
W20260806 03:44:02.880350 5133 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.714s user 0.001s sys 0.000s
I20260806 03:44:03.015056 664 raft_consensus.cc:1226] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: [5.2-5.2]
W20260806 03:44:02.587138 5086 log.cc:924] Time spent T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.458s user 0.000s sys 0.001s
W20260806 03:44:02.360175 4931 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.008s sys 0.003s
W20260806 03:44:02.331285 5009 consensus_meta.cc:287] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.173s user 0.007s sys 0.015s
I20260806 03:44:03.145252 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:02.608357 4885 raft_consensus.cc:3032] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Stepping down as leader of term 7
W20260806 03:44:02.132298 5054 log.cc:924] Time spent T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20260806 03:44:02.949506 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 73476.5us
I20260806 03:44:02.103087 5011 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:03.014981 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:03.015425 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:03.130115 4451 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:02.034302 4473 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.132284 4983 raft_consensus.cc:3032] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:03.132347 4904 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:03.133023 667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 416) took 2283 ms (client timeout 2309 ms). Trace:
I20260806 03:44:03.133196 5139 raft_consensus.cc:2741] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 03:44:02.089422 5105 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.134963 5135 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:44:03.136802 5145 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:02.168583 5116 log.cc:924] Time spent T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
W20260806 03:44:02.881851 5083 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.781s user 0.000s sys 0.002s
I20260806 03:44:03.098835 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:02.766137 5130 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:02.676179 4922 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:02.979725 5123 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:02.531306 4761 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:02.880591 3902 consensus_meta.cc:287] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.126s user 0.003s sys 0.017s
I20260806 03:44:02.979411 5122 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:02.544711 4963 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:02.320394 5026 raft_consensus.cc:686] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:02.801844 4405 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:02.531191 5090 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 03:44:03.192102 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.001s sys 0.008s
I20260806 03:44:03.158262 5035 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:02.654345 4953 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:02.980032 5100 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:44:03.204676 1622 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.309s (SENT) [suppressed 53 similar messages]
I20260806 03:44:03.207670 1665 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:02.193861 5125 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:44:03.196998 667 rpcz_store.cc:269] 0806 03:44:00.850077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.850347 (+ 270us) service_pool.cc:227] Handling call
0806 03:44:03.132853 (+2282506us) spinlock_profiling.cc:243] Waited 53 ms on lock 0x7b54000b2a54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.132965 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121929600}
I20260806 03:44:02.603076 5131 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.134899 768 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1586) took 1938 ms. Trace:
W20260806 03:44:03.234191 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 251089us
I20260806 03:44:03.132050 5134 consensus_queue.cc:1046] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.185s
I20260806 03:44:02.159175 5074 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.249025 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:02.166795 4907 consensus_queue.cc:1035] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 1.043s
I20260806 03:44:03.163044 4885 raft_consensus.cc:729] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:03.216127 4451 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.284192 561 raft_consensus.cc:2454] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:44:03.209856 4983 raft_consensus.cc:729] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:02.331704 4658 consensus_queue.cc:237] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:02.102411 5034 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:03.280220 4841 consensus_meta.cc:287] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.953s user 0.016s sys 0.016s
I20260806 03:44:03.263721 5080 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:02.102224 5043 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:03.286270 5128 log.cc:924] Time spent T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20260806 03:44:03.158411 5009 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:03.137691 4931 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.297s user 0.011s sys 0.006s
I20260806 03:44:03.208245 5011 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:03.340118 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.957s user 0.001s sys 0.009s
I20260806 03:44:03.287580 5130 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.357151 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1407) took 2649 ms. Trace:
I20260806 03:44:03.323495 5122 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.358933 768 rpcz_store.cc:276] 0806 03:44:01.196385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.196593 (+ 208us) service_pool.cc:227] Handling call
0806 03:44:01.197144 (+ 551us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.254615 (+ 57471us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.254659 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.268473 (+ 13814us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.655439 (+386966us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b4800160158. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:44:01.701549 (+ 46110us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:44:01.701590 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.134594 (+1433004us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b5400055fd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:03.134646 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.134687 (+ 41us) raft_consensus.cc:1671] finished
0806 03:44:03.134699 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.134841 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13823,"raft.queue_time_us":14464,"raft.run_cpu_time_us":1764,"raft.run_wall_time_us":398328,"spinlock_wait_cycles":109681152,"thread_start_us":286307,"threads_started":1,"wal-append.queue_time_us":439791,"wal-append.run_cpu_time_us":3891,"wal-append.run_wall_time_us":1549058}
I20260806 03:44:03.314167 4761 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.374994 661 raft_consensus.cc:2396] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 03:44:02.205384 4832 consensus_queue.cc:237] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.295285 5138 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.215514 4904 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.365161 4922 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:03.313817 5153 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.360545 5125 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:03.395951 767 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57212 (request call id 2385) took 1396 ms. Trace:
W20260806 03:44:03.400667 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 196010us
I20260806 03:44:03.278716 5145 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.362390 5139 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:03.000178 5094 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.424628 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1589) took 2147 ms. Trace:
I20260806 03:44:03.425585 562 rpcz_store.cc:276] 0806 03:44:00.707642 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.229518 (+521876us) service_pool.cc:227] Handling call
0806 03:44:01.242546 (+ 13028us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.256116 (+ 13570us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.256159 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.264121 (+ 7962us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.673377 (+409256us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:44:01.686096 (+ 12719us) log.cc:842] Serialized 14 byte log entry
0806 03:44:01.686307 (+ 211us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.686415 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.356873 (+1670458us) raft_consensus.cc:1671] finished
0806 03:44:03.356902 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.357079 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13101,"prepare.queue_time_us":12505,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":78221,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":19074,"thread_start_us":364041,"threads_started":1,"wal-append.queue_time_us":485005}
I20260806 03:44:03.396862 4841 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:44:03.415740 4851 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.671s user 0.015s sys 0.006s
I20260806 03:44:03.446103 547 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
I20260806 03:44:03.391263 4885 consensus_queue.cc:260] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.450665 5134 raft_consensus.cc:3032] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:03.295591 5123 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.449667 4761 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:03.483431 4119 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.355s user 0.005s sys 0.008s
I20260806 03:44:03.456028 767 rpcz_store.cc:276] 0806 03:44:01.999250 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.999396 (+ 146us) service_pool.cc:227] Handling call
0806 03:44:01.999740 (+ 344us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:02.018112 (+ 18372us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:44:03.030255 (+1012143us) log.cc:842] Serialized 14 byte log entry
0806 03:44:03.395241 (+364986us) log.cc:842] Serialized 14 byte log entry
0806 03:44:03.395488 (+ 247us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:03.395560 (+ 72us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:03.395628 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.395714 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.395870 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21045,"prepare.queue_time_us":949505,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":99,"raft.queue_time_us":273870,"raft.run_cpu_time_us":463,"raft.run_wall_time_us":22243,"spinlock_wait_cycles":18841984,"thread_start_us":312287,"threads_started":4,"wal-append.queue_time_us":357803}
I20260806 03:44:03.453123 5141 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.455176 5034 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.455121 5185 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:03.486202 563 raft_consensus.cc:595] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:03.431203 4931 raft_consensus.cc:482] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.480734 773 rpcz_store.cc:276] 0806 03:44:01.259635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.259785 (+ 150us) service_pool.cc:227] Handling call
0806 03:44:01.268218 (+ 8433us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.606545 (+338327us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.606594 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.613850 (+ 7256us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.798536 (+184686us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:01.798573 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.798648 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.406319 (+1607671us) raft_consensus.cc:1671] finished
0806 03:44:03.406346 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.406549 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":145376,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":404518,"threads_started":2,"wal-append.queue_time_us":232828}
I20260806 03:44:03.454844 5181 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:03.490613 674 raft_consensus.cc:1226] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:03.405565 5154 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:03.480908 4819 consensus_meta.cc:287] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.679s user 0.021s sys 0.003s
I20260806 03:44:03.391814 4451 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.372177 5131 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.453521 4963 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.455649 5179 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.584558 5081 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.455361 5153 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.585405 5179 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:03.586927 663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2429) took 2919 ms (client timeout 1522 ms). Trace:
W20260806 03:44:03.587414 663 rpcz_store.cc:269] 0806 03:44:00.667845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.668599 (+ 754us) service_pool.cc:227] Handling call
0806 03:44:03.586662 (+2918063us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b540036c9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.586838 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1014355328}
I20260806 03:44:03.446435 4983 consensus_queue.cc:260] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.396188 5080 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:03.576908 766 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2375) took 2685 ms (client timeout 1986 ms). Trace:
I20260806 03:44:03.579022 4431 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:03.604359 4431 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:03.605899 4431 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.607864 4431 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:03.609957 5074 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.611956 5110 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.612792 5110 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.491586 4907 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:03.582788 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2443) took 2135 ms. Trace:
I20260806 03:44:03.614670 668 rpcz_store.cc:276] 0806 03:44:01.447860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.448036 (+ 176us) service_pool.cc:227] Handling call
0806 03:44:01.448423 (+ 387us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 03:44:01.450459 (+ 2036us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:44:01.494435 (+ 43976us) log.cc:842] Serialized 14 byte log entry
0806 03:44:01.658557 (+164122us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x7b1800189ca0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa657b 00007f321632988a
0806 03:44:02.970527 (+1311970us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 03:44:03.000991 (+ 30464us) log.cc:842] Serialized 78 byte log entry
0806 03:44:03.001172 (+ 181us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:03.001219 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.001293 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.529029 (+527736us) raft_consensus.cc:1671] finished
0806 03:44:03.582547 (+ 53518us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.582709 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3969,"prepare.queue_time_us":31561,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":148607,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":25953,"spinlock_wait_cycles":9957120,"thread_start_us":1311683,"threads_started":2,"wal-append.queue_time_us":654912}
I20260806 03:44:03.616325 4904 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.617439 4907 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.492118 4851 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:03.620452 4761 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.502202 4885 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:03.526698 5009 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.525511 5130 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:03.527947 5143 log.cc:924] Time spent T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
I20260806 03:44:03.588421 5153 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.460206 4922 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.492345 5105 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.453598 5138 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.455885 5125 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:03.443691 5026 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:03.600991 766 rpcz_store.cc:269] 0806 03:44:00.862064 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.863681 (+ 1617us) service_pool.cc:227] Handling call
0806 03:44:03.547094 (+2683413us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54001031d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.547275 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98637056}
I20260806 03:44:03.603570 563 raft_consensus.cc:603] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:44:03.529598 665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2435) took 2855 ms. Trace:
I20260806 03:44:03.606011 5094 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:03.603755 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 441580us
I20260806 03:44:03.648067 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1417) took 2835 ms. Trace:
I20260806 03:44:03.456674 5122 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.607527 5051 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:03.763285 547 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1919) took 3220 ms (client timeout 1672 ms). Trace:
W20260806 03:44:03.798492 547 rpcz_store.cc:269] 0806 03:44:00.542963 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.543934 (+ 971us) service_pool.cc:227] Handling call
0806 03:44:03.762994 (+3219060us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54000221d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.763176 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6671929728}
I20260806 03:44:03.487426 5139 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.738878 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1596) took 2018 ms. Trace:
I20260806 03:44:03.802495 772 rpcz_store.cc:276] 0806 03:44:01.720615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.723084 (+ 2469us) service_pool.cc:227] Handling call
0806 03:44:01.730677 (+ 7593us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:01.986739 (+256062us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.986784 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:01.986841 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:01.986887 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.738610 (+1751723us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b5400184554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:03.738662 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.738810 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":586972544}
W20260806 03:44:03.701587 548 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 440) took 3023 ms (client timeout 2977 ms). Trace:
W20260806 03:44:03.804785 548 rpcz_store.cc:269] 0806 03:44:00.677913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.954638 (+276725us) service_pool.cc:227] Handling call
0806 03:44:03.701302 (+2746664us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540030f054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.701491 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2456791424}
W20260806 03:44:03.757879 561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1942) took 1950 ms (client timeout 1526 ms). Trace:
I20260806 03:44:03.491622 5134 raft_consensus.cc:729] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:03.579424 5123 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.589391 4983 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:03.736028 5174 log.cc:924] Time spent T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.177s user 0.000s sys 0.001s
I20260806 03:44:03.764945 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 515) took 3218 ms. Trace:
I20260806 03:44:03.804924 665 rpcz_store.cc:276] 0806 03:44:00.673886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.685824 (+ 11938us) service_pool.cc:227] Handling call
0806 03:44:03.529348 (+2843524us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b540015a9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.529523 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1672087936}
I20260806 03:44:03.804121 558 rpcz_store.cc:276] 0806 03:44:00.812921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.089378 (+1276457us) service_pool.cc:227] Handling call
0806 03:44:02.089735 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:02.090079 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:02.090109 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:02.090600 (+ 491us) log.cc:842] Serialized 26 byte log entry
0806 03:44:03.009582 (+918982us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:03.194935 (+185353us) log.cc:842] Serialized 14 byte log entry
0806 03:44:03.209116 (+ 14181us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.209224 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.647777 (+438553us) raft_consensus.cc:1671] finished
0806 03:44:03.647812 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.647999 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10975,"prepare.queue_time_us":346132,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":265895,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":42807,"spinlock_wait_cycles":8669952,"thread_start_us":228033,"threads_started":2,"wal-append.queue_time_us":1509241}
I20260806 03:44:03.851989 1662 rpcz_store.cc:276] 0806 03:44:00.546847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.547295 (+ 448us) service_pool.cc:227] Handling call
0806 03:44:00.597987 (+ 50692us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.694718 (+1096731us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.764454 (+2069736us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b540044edd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:03.764855 (+ 401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1867404416}
I20260806 03:44:03.853391 558 raft_consensus.cc:1261] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 03:44:03.589421 5179 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.584628 5168 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.833226 564 consensus_queue.cc:260] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.702214 5110 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.546962 4473 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:03.699410 5113 log.cc:924] Time spent T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 1.352s user 0.008s sys 0.003s
I20260806 03:44:03.838266 548 raft_consensus.cc:1261] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:44:03.800683 547 raft_consensus.cc:1226] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:03.525228 5163 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:03.508288 5164 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.654850 5145 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.876277 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1423) took 2387 ms. Trace:
W20260806 03:44:03.707028 5126 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.273s user 0.000s sys 0.002s
W20260806 03:44:03.696558 5096 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 1.425s user 0.009s sys 0.003s
I20260806 03:44:03.707458 4451 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.687661 4119 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:03.802832 5051 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:44:03.895021 727 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.874s user 0.000s sys 0.025s
I20260806 03:44:03.710093 5011 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.865978 5168 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.652776 4819 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.651273 5035 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.742545 4861 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.902448 769 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1571) took 3132 ms. Trace:
W20260806 03:44:03.890977 5100 consensus_meta.cc:287] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.523s user 0.012s sys 0.001s
W20260806 03:44:03.870134 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 72322.3us
I20260806 03:44:03.710490 4907 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.746093 5080 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.709395 5070 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.836759 5043 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.632843 5154 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.880235 5164 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.648471 5181 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.918212 4819 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.920058 5164 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.650563 4931 raft_consensus.cc:504] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.881731 5163 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:44:03.904376 5134 consensus_queue.cc:260] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:03.629693 4963 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.711086 5186 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:03.925161 5163 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.685887 5141 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.924808 5186 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.869231 5139 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.742653 5131 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.705168 4922 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:03.648777 5185 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.914743 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:03.931758 1662 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:03.930263 5141 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:03.932359 5138 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:03.674342 5081 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.951665 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:03.901317 4119 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.912243 5100 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:03.877583 547 rpcz_store.cc:276] 0806 03:44:01.488922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.800044 (+2311122us) service_pool.cc:227] Handling call
0806 03:44:03.800445 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:03.875951 (+ 75506us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.876198 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:03.873195 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1424) took 2381 ms. Trace:
W20260806 03:44:03.806165 561 rpcz_store.cc:269] 0806 03:44:01.807107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.845714 (+ 38607us) service_pool.cc:227] Handling call
0806 03:44:03.757592 (+1911878us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540011e4d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:03.757760 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3035100928}
I20260806 03:44:03.853816 558 pending_rounds.cc:76] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Aborting all ops after (but not including) 6
W20260806 03:44:03.898736 727 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.000s sys 0.025s
I20260806 03:44:03.903580 769 rpcz_store.cc:276] 0806 03:44:00.770389 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.770542 (+ 153us) service_pool.cc:227] Handling call
0806 03:44:00.770861 (+ 319us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:00.771200 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:00.771230 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.633002 (+861772us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b4800160158. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:44:01.640119 (+ 7117us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b2c00148c38. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f3214d02e11 00007f3214d12442 00007f321635ac11 00007f321631f58a 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772
0806 03:44:01.640745 (+ 626us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.804297 (+163552us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:44:01.804337 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.804411 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.621458 (+1817047us) spinlock_profiling.cc:243] Waited 9.86 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:44:03.902208 (+280750us) raft_consensus.cc:1671] finished
0806 03:44:03.902231 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.902365 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":3969,"raft.run_wall_time_us":861619,"spinlock_wait_cycles":33688448,"thread_start_us":163088,"threads_started":1,"wal-append.queue_time_us":413854}
I20260806 03:44:03.961743 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:03.903869 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1587) took 2677 ms. Trace:
W20260806 03:44:04.034576 546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.892s user 0.013s sys 0.038s
W20260806 03:44:04.040987 502 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.264s user 0.002s sys 0.023s
I20260806 03:44:03.982930 5185 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:04.111616 502 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.347s user 0.002s sys 0.024s
I20260806 03:44:04.042495 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1418) took 3229 ms. Trace:
I20260806 03:44:04.120328 5185 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260806 03:44:04.058305 1665 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:04.131413 4931 leader_election.cc:288] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.124109 542 rpcz_store.cc:276] 0806 03:44:00.813217 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.093714 (+1280497us) service_pool.cc:227] Handling call
0806 03:44:02.096972 (+ 3258us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:02.097732 (+ 760us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:02.097772 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:02.962719 (+864947us) log.cc:842] Serialized 26 byte log entry
0806 03:44:03.009047 (+ 46328us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:03.246908 (+237861us) log.cc:842] Serialized 14 byte log entry
0806 03:44:03.247150 (+ 242us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.247246 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.745488 (+498242us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b4800150258. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321b1c67f1 00007f321b1bd8e7 00007f321b1bd77a 00007f321b2b195f 00007f321b2b1627 00007f321b2c5b2c 00007f321b2c5ad8 00007f321b2c5a58 00007f321b2c5a13 00007f321b2c4cbc 00007f321a246be6
0806 03:44:04.041289 (+295801us) raft_consensus.cc:1671] finished
0806 03:44:04.041538 (+ 249us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.042084 (+ 546us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35535,"prepare.queue_time_us":286404,"prepare.run_cpu_time_us":3167,"prepare.run_wall_time_us":13025,"raft.queue_time_us":211795,"raft.run_cpu_time_us":17753,"raft.run_wall_time_us":1610596,"spinlock_wait_cycles":23045504,"thread_start_us":283220,"threads_started":2,"wal-append.queue_time_us":916198}
I20260806 03:44:04.136384 542 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260806 03:44:04.137101 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 455) took 2160 ms. Trace:
I20260806 03:44:04.137429 542 rpcz_store.cc:276] 0806 03:44:01.976496 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.134980 (+2158484us) service_pool.cc:227] Handling call
0806 03:44:04.135694 (+ 714us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:04.136799 (+ 1105us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.137012 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:03.920054 5181 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.151429 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2445) took 1155 ms. Trace:
I20260806 03:44:04.151929 674 rpcz_store.cc:276] 0806 03:44:02.995752 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.995910 (+ 158us) service_pool.cc:227] Handling call
0806 03:44:03.098420 (+102510us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:03.603078 (+504658us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.150703 (+547625us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b54000668d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:04.151150 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":895448960}
I20260806 03:44:04.058292 776 rpcz_store.cc:276] 0806 03:44:01.226788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.230615 (+ 3827us) service_pool.cc:227] Handling call
0806 03:44:01.230965 (+ 350us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:01.241498 (+ 10533us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:01.241555 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:01.253527 (+ 11972us) log.cc:842] Serialized 26 byte log entry
0806 03:44:01.498635 (+245108us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:01.498692 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:01.498798 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:03.617098 (+2118300us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:44:03.900719 (+283621us) raft_consensus.cc:1671] finished
0806 03:44:03.900753 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:03.903799 (+ 3046us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23082,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8759,"spinlock_wait_cycles":28142464,"thread_start_us":168835,"threads_started":1,"wal-append.queue_time_us":763567}
I20260806 03:44:04.157090 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43080 (request call id 24) took 2695 ms. Trace:
I20260806 03:44:04.157574 563 rpcz_store.cc:276] 0806 03:44:01.461173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.627747 (+166574us) service_pool.cc:227] Handling call
0806 03:44:04.156537 (+2528790us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54000221d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f3216319bc3 00007f321b26381a 00007f3214956d04 00007f3214956c92 00007f3214956bc1 00007f3214956b3c 00007f3214955df2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:04.156797 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4122456960}
I20260806 03:44:04.159403 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.120615 1662 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:04.160559 772 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 10.
I20260806 03:44:04.172571 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.176069 766 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:04.176556 4963 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.178745 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.179674 776 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 5.
I20260806 03:44:04.180573 5120 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.181432 5120 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.184206 5120 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.193039 564 raft_consensus.cc:3037] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:44:04.058925 546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.916s user 0.013s sys 0.048s
I20260806 03:44:03.985811 5186 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.061669 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 454) took 2085 ms. Trace:
I20260806 03:44:03.984553 5154 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:04.046700 616 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.006s sys 0.017s
I20260806 03:44:04.062299 558 pending_rounds.cc:97] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 03:44:04.062433 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:04.242692 561 rpcz_store.cc:276] 0806 03:44:01.976201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.059765 (+2083564us) service_pool.cc:227] Handling call
0806 03:44:04.060561 (+ 796us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:04.061021 (+ 460us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.061057 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:04.061154 (+ 97us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:04.061219 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.061309 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.061448 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:04.058928 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.063126 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:04.284789 616 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.893s user 0.006s sys 0.018s
I20260806 03:44:04.062647 548 rpcz_store.cc:276] 0806 03:44:01.491889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.825571 (+2333682us) service_pool.cc:227] Handling call
0806 03:44:03.837840 (+ 12269us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:03.872734 (+ 34894us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:03.873107 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:04.308020 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.314173 4934 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:04.314571 618 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.038s sys 0.096s
W20260806 03:44:04.315095 618 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.930s user 0.038s sys 0.096s
W20260806 03:44:04.315900 1657 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.018s sys 0.039s
W20260806 03:44:04.316378 1657 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s user 0.018s sys 0.039s
W20260806 03:44:04.318136 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.31536e+06us
I20260806 03:44:04.327219 1662 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:04.329288 727 leader_election.cc:398] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:04.283430 558 raft_consensus.cc:2866] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:44:04.330763 4119 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.331053 558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35214 (request call id 1941) took 2551 ms. Trace:
W20260806 03:44:04.332079 499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.009s sys 0.026s
W20260806 03:44:04.332523 499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.048s user 0.010s sys 0.027s
W20260806 03:44:04.332273 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 952655us
I20260806 03:44:04.335129 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1425) took 2329 ms. Trace:
I20260806 03:44:04.335561 542 rpcz_store.cc:276] 0806 03:44:01.995757 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.138321 (+2142564us) service_pool.cc:227] Handling call
0806 03:44:04.238044 (+ 99723us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:04.307861 (+ 69817us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.307904 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:04.318461 (+ 10557us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:04.324673 (+ 6212us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.324756 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.324918 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:04.337219 614 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.007s sys 0.015s
W20260806 03:44:04.337576 614 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.007s sys 0.015s
I20260806 03:44:04.338182 5081 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.339377 5198 raft_consensus.cc:482] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:04.340008 1671 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:44:04.340837 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.340320 5198 raft_consensus.cc:504] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.343842 766 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:04.345389 5198 leader_election.cc:288] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.331707 558 rpcz_store.cc:276] 0806 03:44:01.779840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.852567 (+2072727us) service_pool.cc:227] Handling call
0806 03:44:03.853031 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:04.330554 (+477523us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.330946 (+ 392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:04.360813 5168 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.362098 564 raft_consensus.cc:1261] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260806 03:44:04.366940 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 79287us
I20260806 03:44:04.367784 3902 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.368661 3902 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.370304 776 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
W20260806 03:44:04.391552 4906 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260806 03:44:04.393187 3902 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.393474 5190 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.394117 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1395) took 3854 ms. Trace:
I20260806 03:44:04.394613 564 rpcz_store.cc:276] 0806 03:44:00.539857 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:01.079077 (+539220us) service_pool.cc:227] Handling call
0806 03:44:01.079465 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:03.136192 (+2056727us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b5400234c20. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32162b5951 00007f3216316321 00007f3216317b4e 00007f3216321c89 00007f3216322556 00007f321631eedf 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612
0806 03:44:03.771453 (+635261us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400234c20. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32162b4dc0 00007f321631633a 00007f3216317b4e 00007f3216321c89 00007f3216322556 00007f321631eedf 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612
0806 03:44:04.121667 (+350214us) spinlock_profiling.cc:243] Waited 61.2 ms on lock 0x7b5400234c20. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32162acf87 00007f3216316367 00007f3216317b4e 00007f3216321c89 00007f3216322556 00007f321631eedf 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612
0806 03:44:04.363986 (+242319us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.394025 (+ 30039us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1371717376}
I20260806 03:44:04.402904 5147 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.409021 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1426) took 2413 ms. Trace:
I20260806 03:44:04.409525 563 rpcz_store.cc:276] 0806 03:44:01.996056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.198907 (+2202851us) service_pool.cc:227] Handling call
0806 03:44:04.341654 (+142747us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:04.408592 (+ 66938us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.408621 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:04.408700 (+ 79us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:04.408744 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.408804 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.408944 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:04.409054 5112 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260806 03:44:04.414773 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.414299 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.416074 1665 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:04.416182 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.417910 5100 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.419040 4405 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.419660 4405 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.421221 5154 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.421995 5154 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.424997 5154 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.315299 4934 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.429374 4934 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:04.430054 5146 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:04.430696 615 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:04.430171 4757 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.431306 4431 consensus_queue.cc:1046] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260806 03:44:04.431216 5146 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.434830 5198 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:04.452756 5198 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:04.403775 5147 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.402194 4786 consensus_queue.cc:1035] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260806 03:44:04.451329 4757 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.453290 5198 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:04.453303 376 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet 90b8bbd25b8646eb93f739aafb4d00f8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d8469561a52c4bbc81dce786f1211ee6 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:44:04.469560 5051 consensus_meta.cc:287] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.566s user 0.018s sys 0.005s
I20260806 03:44:04.470434 4757 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.471050 5146 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.475335 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.475924 5147 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.476343 769 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:04.478554 5134 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:04.482045 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 153108us
I20260806 03:44:04.498868 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.499284 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:04.499522 4983 consensus_meta.cc:287] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.642s user 0.010s sys 0.008s
I20260806 03:44:04.499958 1665 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:44:04.502020 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.502482 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.502646 1670 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:04.503965 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.506827 4405 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.394550 5190 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.516305 1671 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:04.530958 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.470124 5051 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:04.515889 4829 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.000s sys 0.020s
W20260806 03:44:04.538295 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 106263us
I20260806 03:44:04.513336 5190 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.524936 727 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:04.540585 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.522128 4906 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:44:04.548580 4829 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.118s user 0.001s sys 0.022s
I20260806 03:44:04.559161 1671 raft_consensus.cc:2396] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260806 03:44:04.560573 728 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.319s user 0.057s sys 0.165s
W20260806 03:44:04.566354 728 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.325s user 0.057s sys 0.165s
I20260806 03:44:04.567634 4786 raft_consensus.cc:3032] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:44:04.567497 4906 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.568755 769 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:04.570015 5204 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:04.572566 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.573781 5154 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.568233 4786 raft_consensus.cc:729] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:04.572376 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.573972 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.557545 5147 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:04.557322 4983 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.574008 5191 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.596032 499 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:04.585268 4906 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.602802 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:04.628803 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.630471 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:04.631557 674 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:04.632800 5204 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:04.633196 770 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.095s user 0.010s sys 0.014s
I20260806 03:44:04.635573 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:04.633729 770 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.176s user 0.010s sys 0.015s
W20260806 03:44:04.634995 501 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.237s user 0.019s sys 0.006s
I20260806 03:44:04.633638 499 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:04.651674 501 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.019s sys 0.007s
W20260806 03:44:04.652081 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 132109us
W20260806 03:44:04.647194 562 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s user 0.008s sys 0.022s
I20260806 03:44:04.594060 5147 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.633611 5204 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:04.638808 1625 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.003s sys 0.040s
W20260806 03:44:04.630467 5083 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.173s user 0.006s sys 0.023s
I20260806 03:44:04.596180 5154 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:04.664216 562 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s user 0.008s sys 0.022s
W20260806 03:44:04.667977 5083 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.282s user 0.006s sys 0.024s
W20260806 03:44:04.668226 544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s user 0.022s sys 0.040s
W20260806 03:44:04.668768 544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s user 0.022s sys 0.040s
I20260806 03:44:04.669277 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:04.647068 4658 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.929s user 0.020s sys 0.020s
I20260806 03:44:04.670342 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:04.670600 5154 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.671514 667 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:44:04.670626 4658 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.047s user 0.020s sys 0.022s
I20260806 03:44:04.671762 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 651) took 1670 ms. Trace:
I20260806 03:44:04.672767 5154 raft_consensus.cc:482] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:04.673607 5154 raft_consensus.cc:504] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:04.674388 5021 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.018s sys 0.029s
I20260806 03:44:04.674593 4405 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:04.674857 5021 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s user 0.018s sys 0.029s
I20260806 03:44:04.675350 4405 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.676146 5154 leader_election.cc:288] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.677832 4405 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.657606 615 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:04.670012 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:04.638046 4906 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:04.665565 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 523) took 1822 ms. Trace:
I20260806 03:44:04.604440 4786 consensus_queue.cc:260] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.672834 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:04.692687 498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.753s user 0.011s sys 0.024s
I20260806 03:44:04.670185 5206 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:44:04.698097 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 181158us
I20260806 03:44:04.637948 5191 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:04.660733 4909 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.292s user 0.015s sys 0.043s
W20260806 03:44:04.678522 1625 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.167s user 0.004s sys 0.040s
I20260806 03:44:04.650357 4829 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:44:04.700889 498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.829s user 0.011s sys 0.025s
W20260806 03:44:04.697670 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 74586.4us
I20260806 03:44:04.680071 5081 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:04.622741 4983 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.689898 1657 rpcz_store.cc:276] 0806 03:44:02.835993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.836190 (+ 197us) service_pool.cc:227] Handling call
0806 03:44:02.836643 (+ 453us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:03.394536 (+557893us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:03.394574 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:03.395017 (+ 443us) log.cc:842] Serialized 26 byte log entry
0806 03:44:04.317039 (+922022us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:04.317171 (+ 132us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.339333 (+ 22162us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54004506d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:04.339396 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:04.633886 (+294490us) raft_consensus.cc:1671] finished
0806 03:44:04.633908 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.658220 (+ 24312us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":188859,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":1282100096,"thread_start_us":921492,"threads_started":1,"wal-append.queue_time_us":1235472}
I20260806 03:44:04.698127 5206 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:04.713433 4909 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.466s user 0.015s sys 0.044s
I20260806 03:44:04.736943 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:04.728703 1624 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:04.719326 4934 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:44:04.672281 770 rpcz_store.cc:276] 0806 03:44:03.001241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.002816 (+ 1575us) service_pool.cc:227] Handling call
0806 03:44:03.003213 (+ 397us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:03.456943 (+453730us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 03:44:03.457274 (+ 331us) log.cc:842] Serialized 14 byte log entry
0806 03:44:04.662162 (+1204888us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:04.670608 (+ 8446us) raft_consensus.cc:1631] Marking committed up to 9
0806 03:44:04.670680 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.671416 (+ 736us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b5400312254. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:04.671467 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.671686 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":664,"prepare.queue_time_us":661,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":16796,"spinlock_wait_cycles":1042870656,"thread_start_us":1180136,"threads_started":1}
W20260806 03:44:04.736753 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 918628us
I20260806 03:44:04.701457 5048 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:04.755525 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.760089 663 raft_consensus.cc:2396] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 03:44:04.754616 615 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:04.737102 5206 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:04.740087 4906 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:04.767848 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:04.772933 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 85979us
I20260806 03:44:04.775410 4786 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:04.778504 5193 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:04.779062 5193 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:04.779441 5193 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:04.782462 5026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.033s sys 0.051s
W20260806 03:44:04.783007 5026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s user 0.033s sys 0.051s
I20260806 03:44:04.784482 4906 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:04.785010 4906 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:04.785705 5191 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.750070 5204 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:04.786556 4832 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.016s sys 0.041s
W20260806 03:44:04.786978 4832 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s user 0.016s sys 0.041s
I20260806 03:44:04.787791 4832 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:04.788228 4832 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:04.788565 4832 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:04.790498 4772 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.042s sys 0.038s
W20260806 03:44:04.790895 4772 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.630s user 0.042s sys 0.038s
I20260806 03:44:04.774252 1624 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:04.794239 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:04.758281 4934 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:04.791958 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:04.804610 5221 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:04.805128 5221 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:04.805491 5221 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:04.769451 5051 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:04.809381 1624 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:44:04.811492 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 106584us
I20260806 03:44:04.807268 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 451) took 3488 ms. Trace:
I20260806 03:44:04.735953 5081 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:04.815963 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 88271.7us
I20260806 03:44:04.818979 5100 raft_consensus.cc:2781] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:04.819195 4431 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.819525 5100 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:04.819993 5100 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:04.820431 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.821396 772 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:44:04.802263 4934 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:44:04.826473 5181 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:04.830641 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:04.751700 4829 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.835125 768 raft_consensus.cc:3037] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:04.841095 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.813330 562 rpcz_store.cc:276] 0806 03:44:01.319031 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.481396 (+2162365us) service_pool.cc:227] Handling call
0806 03:44:03.492353 (+ 10957us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:03.494904 (+ 2551us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:03.494946 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:03.495496 (+ 550us) log.cc:842] Serialized 26 byte log entry
0806 03:44:04.765636 (+1270140us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:04.771644 (+ 6008us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.798531 (+ 26887us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:04.798601 (+ 70us) raft_consensus.cc:1671] finished
0806 03:44:04.798621 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:04.807156 (+ 8535us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2099,"raft.queue_time_us":127036,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164,"thread_start_us":1035528,"threads_started":1,"wal-append.queue_time_us":1220853}
I20260806 03:44:04.856983 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.857882 1658 raft_consensus.cc:2379] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:04.858269 4906 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.858790 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.859542 766 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 5.
I20260806 03:44:04.860435 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.860522 4906 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.861408 766 raft_consensus.cc:2396] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:44:04.822356 4119 raft_consensus.cc:2781] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:04.864836 1624 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:04.840301 5147 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.867175 768 raft_consensus.cc:2454] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:04.838186 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:04.883427 5207 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:04.884207 5207 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:04.887117 5207 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:04.894981 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:04.899835 674 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:44:04.900985 667 raft_consensus.cc:1261] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260806 03:44:04.819993 4431 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:04.823014 5048 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.282709 544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1420) took 4420 ms. Trace:
I20260806 03:44:04.902503 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.002461 776 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 3.
W20260806 03:44:05.068066 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 110452us
I20260806 03:44:05.561656 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:44:05.204151 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 463738us
I20260806 03:44:05.563252 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:04.921990 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 90483.3us
W20260806 03:44:05.034125 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 119165us
I20260806 03:44:05.570307 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:04.901491 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:05.079358 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 46004.4us
I20260806 03:44:04.998421 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:05.573087 1663 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:44:05.299337 548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.060s sys 0.107s
W20260806 03:44:05.576004 548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.166s user 0.069s sys 0.120s
W20260806 03:44:05.019959 546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.051s sys 0.092s
W20260806 03:44:05.578440 546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.242s user 0.065s sys 0.112s
I20260806 03:44:05.582053 1670 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:44:05.309193 5206 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.533s user 0.012s sys 0.000s
W20260806 03:44:05.545034 4786 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.769s user 0.012s sys 0.002s
I20260806 03:44:05.583684 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.584436 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:04.901953 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:04.998926 5120 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:05.599072 1624 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:05.600385 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:05.601814 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:05.327001 5198 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.833s user 0.012s sys 0.006s
I20260806 03:44:05.169667 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.172803 4829 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.596065 665 raft_consensus.cc:2396] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260806 03:44:04.908313 4983 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:05.576116 5125 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 1.459s user 0.019s sys 0.006s
I20260806 03:44:05.000309 5051 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.556070 5081 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.604311 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.620571 776 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:44:05.605719 5222 log.cc:924] Time spent T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.567s user 0.000s sys 0.001s
W20260806 03:44:05.626992 5227 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260806 03:44:04.904341 4119 raft_consensus.cc:686] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:05.629202 5032 raft_consensus.cc:482] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.630003 5032 raft_consensus.cc:504] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.630241 4119 consensus_queue.cc:237] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.607829 767 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 3.
I20260806 03:44:05.569119 544 rpcz_store.cc:276] 0806 03:44:00.862174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:03.001464 (+2139290us) service_pool.cc:227] Handling call
0806 03:44:03.767280 (+765816us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:03.835087 (+ 67807us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:03.835123 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:03.835797 (+ 674us) log.cc:842] Serialized 26 byte log entry
0806 03:44:04.669368 (+833571us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:44:04.669427 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.669532 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:05.156169 (+486637us) raft_consensus.cc:1671] finished
0806 03:44:05.156198 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:05.227503 (+ 71305us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540011e4d0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:05.282599 (+ 55096us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45494,"raft.queue_time_us":54235,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":1760296704,"thread_start_us":832804,"threads_started":1,"wal-append.queue_time_us":976141}
I20260806 03:44:05.632447 5032 leader_election.cc:288] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:04.901701 5218 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260806 03:44:05.320793 5134 consensus_meta.cc:287] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.842s user 0.009s sys 0.007s
I20260806 03:44:05.606616 772 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:44:05.633725 5032 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:05.079809 5186 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.601226 5206 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.634855 5186 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.635739 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.636656 5206 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.637171 5186 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:05.592113 5191 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.000622 4931 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.107731 4963 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.614607 5125 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.625608 5051 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.626791 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.328619 4909 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.621737 663 raft_consensus.cc:2354] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8.
I20260806 03:44:04.901912 5217 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:44:05.642257 661 consensus_meta.cc:287] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.948s user 0.012s sys 0.001s
I20260806 03:44:05.644546 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.656966 661 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:05.657428 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1807) took 1027 ms. Trace:
I20260806 03:44:05.598526 5120 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:04.957326 5181 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.612784 5147 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:05.646620 727 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:05.660707 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:05.663539 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 68181.1us
I20260806 03:44:05.657745 661 rpcz_store.cc:276] 0806 03:44:04.629652 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.663972 (+ 34320us) service_pool.cc:227] Handling call
0806 03:44:05.657376 (+993404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:05.655017 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:05.668286 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.676364 674 raft_consensus.cc:2354] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8.
I20260806 03:44:05.671847 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:05.679272 4934 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.824s user 0.009s sys 0.004s
I20260806 03:44:05.679401 4953 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.679766 5048 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.679833 4934 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.680362 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.652381 5032 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.682730 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.684031 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.685915 498 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:05.687016 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.687539 498 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:05.687770 772 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:44:05.688848 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 74476.7us
W20260806 03:44:05.689500 4832 consensus_meta.cc:287] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.900s user 0.009s sys 0.003s
W20260806 03:44:05.690774 5233 log.cc:924] Time spent T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.092s user 0.001s sys 0.001s
I20260806 03:44:05.690594 727 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:05.690012 4832 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:05.686164 5221 consensus_meta.cc:287] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.880s user 0.011s sys 0.000s
I20260806 03:44:05.687026 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.682775 5032 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.690440 4405 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:05.697119 1657 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:05.615671 5198 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.696995 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.697896 668 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:05.665853 5120 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:05.649329 5191 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.722836 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:05.724604 5191 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.726069 5181 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.662441 5217 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:05.669059 5147 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:05.728919 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1427) took 3690 ms. Trace:
W20260806 03:44:05.660955 4772 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.008s sys 0.008s
I20260806 03:44:05.715741 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.716013 4829 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:05.716558 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.718149 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.719986 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.720599 5051 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.690673 4934 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.702569 5186 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:05.739760 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:05.731820 5193 consensus_meta.cc:287] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.952s user 0.012s sys 0.004s
W20260806 03:44:05.739384 4772 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.948s user 0.008s sys 0.008s
I20260806 03:44:05.732197 561 rpcz_store.cc:276] 0806 03:44:02.038176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.291924 (+2253748us) service_pool.cc:227] Handling call
0806 03:44:04.332891 (+ 40967us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:04.410991 (+ 78100us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.411027 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:04.411549 (+ 522us) log.cc:842] Serialized 26 byte log entry
0806 03:44:04.754532 (+342983us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:44:04.754571 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:04.754652 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:05.728629 (+973977us) raft_consensus.cc:1671] finished
0806 03:44:05.728666 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:05.728843 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77404,"raft.queue_time_us":78787,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":342481,"threads_started":1,"wal-append.queue_time_us":1132738}
I20260806 03:44:05.662853 4931 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.746649 1663 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 9.
I20260806 03:44:05.747991 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:05.734761 5209 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.958s user 0.039s sys 0.059s
W20260806 03:44:05.759069 5209 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.985s user 0.039s sys 0.059s
I20260806 03:44:05.734640 5147 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:05.753278 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.679937 4953 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.731406 5217 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:05.759106 769 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:05.734946 5125 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:05.705166 5221 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.744942 5218 raft_consensus.cc:482] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.690521 4963 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.764602 772 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:44:05.765434 1662 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 4.
I20260806 03:44:05.768481 767 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:44:05.766456 499 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:05.774080 728 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:05.763077 5193 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.708199 4909 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:05.789013 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 98514.1us
I20260806 03:44:05.771718 5206 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.767907 4405 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.761379 5081 raft_consensus.cc:2741] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260806 03:44:05.804840 5206 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:05.782629 547 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.010s user 0.004s sys 0.015s
W20260806 03:44:05.808351 547 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.125s user 0.005s sys 0.015s
I20260806 03:44:05.784489 5191 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:44:05.768128 5181 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.799788 5051 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:44:05.814006 547 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 453) took 3838 ms. Trace:
I20260806 03:44:05.815774 669 raft_consensus.cc:2354] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7.
W20260806 03:44:05.811275 4911 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.926s user 0.006s sys 0.037s
I20260806 03:44:05.816917 4832 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:05.821529 1622 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.995s user 0.008s sys 0.028s
W20260806 03:44:05.821878 1622 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.047s user 0.008s sys 0.029s
W20260806 03:44:05.822247 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 1.04792e+06us
I20260806 03:44:05.815804 5154 raft_consensus.cc:2726] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:05.815842 5051 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:05.814787 4983 raft_consensus.cc:482] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.815984 4431 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:05.822346 4911 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.051s user 0.007s sys 0.038s
I20260806 03:44:05.840699 727 leader_election.cc:398] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:05.841128 727 leader_election.cc:401] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:44:05.829703 5181 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.847095 5206 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.848695 5181 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.848898 5193 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.850665 727 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:05.851334 5221 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.816095 547 rpcz_store.cc:276] 0806 03:44:01.975843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.056920 (+2081077us) service_pool.cc:227] Handling call
0806 03:44:04.112253 (+ 55333us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:04.177154 (+ 64901us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.177185 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:04.177964 (+ 779us) log.cc:842] Serialized 26 byte log entry
0806 03:44:04.650861 (+472897us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:05.808872 (+1158011us) log.cc:842] Serialized 14 byte log entry
0806 03:44:05.813671 (+ 4799us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:05.813764 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:05.813804 (+ 40us) raft_consensus.cc:1671] finished
0806 03:44:05.813817 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:05.813946 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4526,"prepare.queue_time_us":1161317,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":13653,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":237,"thread_start_us":1585975,"threads_started":2,"wal-append.queue_time_us":1131805}
I20260806 03:44:05.834491 5051 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:44:05.860862 770 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.063s sys 0.065s
W20260806 03:44:05.862984 770 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.650s user 0.063s sys 0.065s
I20260806 03:44:05.864491 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:05.865283 1662 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:05.810889 5218 raft_consensus.cc:504] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.820362 4405 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:05.860500 5100 consensus_meta.cc:287] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 1.040s user 0.007s sys 0.006s
I20260806 03:44:05.852859 5198 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:05.873947 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 273651us
W20260806 03:44:05.874432 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 248539us
I20260806 03:44:05.853031 4983 raft_consensus.cc:504] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.885880 4909 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.886528 4909 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:05.887760 4953 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.888489 4953 consensus_queue.cc:1035] T 0488766430fc45ddb1edb22dd107266d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:44:05.898671 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.903724 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.909776 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 430) took 1009 ms. Trace:
I20260806 03:44:05.911727 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.875633 5100 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.910203 669 rpcz_store.cc:276] 0806 03:44:04.900653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.900888 (+ 235us) service_pool.cc:227] Handling call
0806 03:44:05.909552 (+1008664us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540019b9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:05.909713 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315808512}
I20260806 03:44:05.915318 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.915210 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.916085 5026 consensus_queue.cc:1035] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260806 03:44:05.921101 1657 raft_consensus.cc:2379] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:05.905682 4931 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:05.915318 5238 log.cc:924] Time spent T 5dcd22bae8db4028ba3d0e499cd18e13 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260806 03:44:05.923645 5221 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:05.924266 1671 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
W20260806 03:44:05.924935 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 64601.9us
I20260806 03:44:05.917734 4963 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.939375 5218 leader_election.cc:288] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:05.942747 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.947258 4405 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:05.953673 4983 leader_election.cc:288] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.954852 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:05.955752 1665 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:44:05.965386 1662 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:05.969589 499 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:05.930688 5221 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:05.967972 5246 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:05.972620 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 428) took 1005 ms. Trace:
I20260806 03:44:05.972841 499 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:05.974009 5221 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:05.974640 727 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:05.927042 5217 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.978910 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:05.977339 4405 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:05.983846 4405 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:05.973073 664 rpcz_store.cc:276] 0806 03:44:04.899874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.901257 (+ 1383us) service_pool.cc:227] Handling call
0806 03:44:05.596339 (+695082us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:05.625199 (+ 28860us) raft_consensus.cc:1520] Early marking committed up to index 9
0806 03:44:05.625631 (+ 432us) log.cc:842] Serialized 14 byte log entry
0806 03:44:05.904950 (+279319us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:05.905008 (+ 58us) raft_consensus.cc:1631] Marking committed up to 9
0806 03:44:05.905062 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:05.905137 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:05.905268 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1193,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":4257,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":274632,"threads_started":1,"wal-append.queue_time_us":299760}
I20260806 03:44:05.977674 4983 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:05.987831 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:05.990422 4983 raft_consensus.cc:686] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:05.991003 1658 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 03:44:05.980232 5218 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:05.992549 4983 consensus_queue.cc:237] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.993518 1658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60258 (request call id 934) took 1084 ms. Trace:
I20260806 03:44:05.993949 1658 rpcz_store.cc:276] 0806 03:44:04.909270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.926495 (+ 17225us) service_pool.cc:227] Handling call
0806 03:44:05.591347 (+664852us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:05.992437 (+401090us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:05.993233 (+ 796us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b540069d7d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:05.993458 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":910479616}
I20260806 03:44:05.972877 5246 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:05.996716 769 raft_consensus.cc:2396] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:44:05.986728 4405 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:06.018244 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:06.022790 1622 leader_election.cc:334] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.677s (SENT)
I20260806 03:44:06.022109 548 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1428) took 3930 ms. Trace:
I20260806 03:44:06.027987 1622 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.039772 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:06.011196 5221 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260806 03:44:06.043684 5218 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:06.021605 5100 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:06.058913 5247 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260806 03:44:06.059329 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:06.037271 4829 raft_consensus.cc:2726] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:06.070354 1671 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.072443 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:06.072948 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 56467.2us
I20260806 03:44:06.074296 615 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.075318 5243 raft_consensus.cc:2781] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:06.075850 5243 raft_consensus.cc:686] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:06.076830 5243 consensus_queue.cc:237] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.078437 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.079061 776 raft_consensus.cc:3037] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.078925 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.081136 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.082531 5218 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:06.084816 5218 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:06.087210 776 raft_consensus.cc:2454] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:06.093155 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:06.093824 1665 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.095160 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.048291 4832 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:06.092888 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 450) took 4774 ms. Trace:
W20260806 03:44:06.106772 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 419305us
I20260806 03:44:06.107601 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:06.109249 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 139369us
I20260806 03:44:06.108259 766 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:06.114063 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.114526 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.115343 772 raft_consensus.cc:2396] T 7a9afce5efa4491797ebed2aab67532f P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:06.103650 546 rpcz_store.cc:276] 0806 03:44:01.318705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:02.998700 (+1679995us) service_pool.cc:227] Handling call
0806 03:44:03.136058 (+137358us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:04.334698 (+1198640us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.334755 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:04.335683 (+ 928us) log.cc:842] Serialized 26 byte log entry
0806 03:44:05.584860 (+1249177us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:06.034669 (+449809us) log.cc:842] Serialized 14 byte log entry
0806 03:44:06.039743 (+ 5074us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:06.064695 (+ 24952us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:06.064762 (+ 67us) raft_consensus.cc:1671] finished
0806 03:44:06.064779 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:06.092825 (+ 28046us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30925,"prepare.queue_time_us":245904,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":215696,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":53442,"thread_start_us":2757495,"threads_started":3,"wal-append.queue_time_us":1402337}
I20260806 03:44:06.161422 1671 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
W20260806 03:44:06.161963 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 45994.8us
I20260806 03:44:06.107358 5186 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:06.165704 1624 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:06.177788 5125 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761) with higher term. Message: Invalid argument: T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
W20260806 03:44:06.163190 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 377901us
W20260806 03:44:06.168519 1622 leader_election.cc:334] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.309s (SENT)
W20260806 03:44:06.191960 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 85835.1us
I20260806 03:44:06.120445 768 raft_consensus.cc:2396] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:44:06.193936 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.195379 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.195542 5252 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:06.196252 5252 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:06.196647 5252 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:06.199157 5198 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:06.202929 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.170611 5051 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.217149 5051 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.204669 4405 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.218290 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:06.218997 1671 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:44:06.219384 4405 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.220561 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:06.220922 5242 log.cc:924] Time spent T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.221s user 0.001s sys 0.001s
I20260806 03:44:06.221208 1662 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:06.223064 767 raft_consensus.cc:2354] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5.
I20260806 03:44:06.223634 771 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:06.207355 727 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.224493 5120 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.225737 727 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.227129 727 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.229929 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.230595 1658 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:06.231125 1671 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:44:06.025301 548 rpcz_store.cc:276] 0806 03:44:02.091641 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.328378 (+2236737us) service_pool.cc:227] Handling call
0806 03:44:04.329112 (+ 734us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:04.408711 (+ 79599us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:04.408755 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:04.409482 (+ 727us) log.cc:842] Serialized 26 byte log entry
0806 03:44:05.576791 (+1167309us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:44:06.019870 (+443079us) log.cc:842] Serialized 14 byte log entry
0806 03:44:06.020141 (+ 271us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:06.020216 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:06.020255 (+ 39us) raft_consensus.cc:1671] finished
0806 03:44:06.020264 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:06.022045 (+ 1781us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77427,"prepare.queue_time_us":235685,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1508,"raft.queue_time_us":81676,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":4292,"thread_start_us":1574809,"threads_started":2,"wal-append.queue_time_us":1337011}
I20260806 03:44:06.104991 5026 consensus_queue.cc:1046] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260806 03:44:06.232753 728 leader_election.cc:302] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:06.230690 5255 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:44:06.258188 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 61676.6us
I20260806 03:44:06.213791 5198 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:06.247318 5191 raft_consensus.cc:686] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:06.263777 5191 consensus_queue.cc:237] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.259660 5255 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:06.264971 5255 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:06.261386 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 487096us
I20260806 03:44:06.245904 1665 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:06.271646 498 leader_election.cc:302] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:06.245630 4829 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:06.246830 5051 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:06.261775 1662 raft_consensus.cc:2454] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:06.246837 4934 raft_consensus.cc:2781] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:06.279717 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1620) took 1441 ms. Trace:
I20260806 03:44:06.260092 4909 raft_consensus.cc:2781] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:44:06.257897 5186 raft_consensus.cc:2726] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:06.277870 616 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:06.289542 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 203100us
I20260806 03:44:06.273101 5198 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.230875 5147 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.293272 499 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:06.293632 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 63820.1us
I20260806 03:44:06.260788 5246 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.285173 775 rpcz_store.cc:276] 0806 03:44:04.829761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:05.644491 (+814730us) service_pool.cc:227] Handling call
0806 03:44:05.669172 (+ 24681us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:05.707474 (+ 38302us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:05.707520 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:05.742157 (+ 34637us) log.cc:842] Serialized 26 byte log entry
0806 03:44:06.103520 (+361363us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:06.103557 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:06.103622 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:06.271323 (+167701us) raft_consensus.cc:1671] finished
0806 03:44:06.271347 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:06.271486 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21715,"raft.queue_time_us":33178,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":4574,"thread_start_us":334463,"threads_started":1,"wal-append.queue_time_us":490702}
I20260806 03:44:06.279089 5120 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:06.310091 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.283057 5253 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:06.316811 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:06.317481 547 raft_consensus.cc:3037] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:06.299585 5154 raft_consensus.cc:2726] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:06.318833 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1610) took 1492 ms. Trace:
I20260806 03:44:06.323594 770 rpcz_store.cc:276] 0806 03:44:04.826506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:04.830840 (+ 4334us) service_pool.cc:227] Handling call
0806 03:44:04.876621 (+ 45781us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:05.109061 (+232440us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:44:05.863642 (+754581us) log.cc:842] Serialized 14 byte log entry
0806 03:44:06.318387 (+454745us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:06.318462 (+ 75us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:06.318532 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:06.318619 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:06.318759 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1435,"prepare.queue_time_us":801367,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":6959,"thread_start_us":1066457,"threads_started":2}
I20260806 03:44:06.296591 4934 raft_consensus.cc:686] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:06.326149 546 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:44:06.298184 4909 raft_consensus.cc:686] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:06.340210 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.342705 561 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.346232 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.350659 562 raft_consensus.cc:1226] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:06.322628 5026 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:06.355535 5026 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:06.356096 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57222 (request call id 1619) took 1526 ms. Trace:
I20260806 03:44:06.372998 5198 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:06.511680 5283 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.183s user 0.003s sys 0.000s
I20260806 03:44:06.509045 773 rpcz_store.cc:276] 0806 03:44:04.829458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:05.638905 (+809447us) service_pool.cc:227] Handling call
0806 03:44:05.639350 (+ 445us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:05.682444 (+ 43094us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:05.682477 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:05.683022 (+ 545us) log.cc:842] Serialized 26 byte log entry
0806 03:44:05.931374 (+248352us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18003658c0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133d9345 00007f321c210a39 00007f32133f68bb 00007f32133f4cbf 00007f32133f6e22 00007f3215fa05bb 00007f3215fa6063 00007f3215fa6411 00007f32162f35cf
0806 03:44:06.177469 (+246095us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:06.177520 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:06.177635 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:06.355653 (+178018us) raft_consensus.cc:1671] finished
0806 03:44:06.355675 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:06.355906 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8251,"raft.queue_time_us":10584,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":33950976,"thread_start_us":245884,"threads_started":1,"wal-append.queue_time_us":576573}
W20260806 03:44:06.477506 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 311957us
I20260806 03:44:06.313853 5265 raft_consensus.cc:2781] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:06.361495 5147 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:06.523958 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 292751us
I20260806 03:44:06.492223 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.561793 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.395211 547 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
W20260806 03:44:06.451802 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 126844us
I20260806 03:44:06.469920 5252 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:06.370769 4963 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:44:06.346807 5253 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.356848 4909 consensus_queue.cc:237] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:06.568465 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 51867.9us
I20260806 03:44:06.359194 4934 consensus_queue.cc:237] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.369071 5264 raft_consensus.cc:3032] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:06.574313 616 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.360775 5255 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:06.571269 562 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.542944 5191 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:06.356069 5026 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.668936 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:06.600733 1624 leader_election.cc:398] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5.
I20260806 03:44:06.673724 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:06.670885 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 61013.4us
I20260806 03:44:06.668437 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.603588 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.583786 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.679294 1624 leader_election.cc:401] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:44:06.538892 5217 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:06.683583 5252 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:06.573149 4963 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:06.687695 4963 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.568675 5265 raft_consensus.cc:686] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:06.692278 5026 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.691577 5265 consensus_queue.cc:237] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:06.693297 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 119054us
I20260806 03:44:06.694067 616 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:06.692829 561 raft_consensus.cc:1261] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 03:44:06.685688 727 leader_election.cc:334] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.672s (SENT)
I20260806 03:44:06.679620 548 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.680809 542 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.695971 727 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:06.689785 5255 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:06.696915 547 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 11.
I20260806 03:44:06.698021 5125 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:06.583422 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.700973 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.758059 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.877535 4963 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:44:06.803400 1622 leader_election.cc:334] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.977s (SENT)
W20260806 03:44:06.810736 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 132074us
I20260806 03:44:06.729969 5217 raft_consensus.cc:2726] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937) with higher term. Message: Invalid argument: T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5.
I20260806 03:44:06.677642 5264 raft_consensus.cc:729] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:06.879650 501 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:06.734753 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.876716 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.900279 562 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.649273 403 catalog_manager.cc:5495] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 6, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:06.695786 544 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.884052 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:06.722437 5026 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:44:06.915935 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.918155 772 raft_consensus.cc:2396] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 03:44:06.912757 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.699119 5252 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:44:06.887054 563 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:06.703526 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.877277 548 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:06.888702 558 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.662871 5253 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.881461 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:06.902194 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.646895 5198 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:06.904246 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 100910us
I20260806 03:44:06.904536 771 raft_consensus.cc:1226] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
W20260806 03:44:06.724578 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 146613us
I20260806 03:44:06.926896 561 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:06.931879 5125 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:06.934204 615 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.934738 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.935467 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.935930 615 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:06.927446 564 raft_consensus.cc:3037] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.928828 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:06.712576 5191 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:06.935756 558 raft_consensus.cc:2396] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260806 03:44:06.931456 769 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:06.936851 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.950073 546 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 4.
I20260806 03:44:06.950826 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.937609 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:06.938449 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:06.939615 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.952299 562 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:06.952382 771 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:06.940982 4786 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 03:44:06.946717 727 connection.cc:232] RPC call timeout handler was delayed by -3.249475448159501s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
I20260806 03:44:06.947243 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.911357 5026 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:06.949239 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:06.953557 5147 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:44:06.953770 5287 consensus_queue.cc:1035] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:44:06.954900 5191 leader_election.cc:288] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:06.926353 5252 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:06.957782 727 connection.cc:232] RPC call timeout handler was delayed by -3.2485754482913762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260806 03:44:06.983569 727 connection.cc:232] RPC call timeout handler was delayed by -3.2485754482913762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260806 03:44:06.983978 727 connection.cc:232] RPC call timeout handler was delayed by -3.2206754481885582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20260806 03:44:06.984290 1664 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 03:44:06.984339 727 connection.cc:232] RPC call timeout handler was delayed by -2.9056754482444376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260806 03:44:06.984831 727 connection.cc:232] RPC call timeout handler was delayed by -2.8507754481397569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
I20260806 03:44:06.960693 4909 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:06.967823 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:06.968567 544 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:06.988016 1663 raft_consensus.cc:1226] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 10. Current term is 11. Ops: []
W20260806 03:44:06.970083 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 90517.4us
I20260806 03:44:06.988541 1665 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:44:06.932916 403 catalog_manager.cc:5495] T 2bea9147cd344c6d8e771c5d9968f2e7 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 6 to 7, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 7 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:06.974578 768 raft_consensus.cc:2454] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:06.977427 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.973445 5026 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:06.979628 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:06.985509 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:06.948945 4405 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:06.983105 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:06.987008 564 raft_consensus.cc:2454] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:06.989907 773 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:06.991655 563 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [term 6 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:06.993387 769 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:06.991822 727 connection.cc:232] RPC call timeout handler was delayed by -1.2784754482563585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3715s for a response
I20260806 03:44:06.995333 1671 raft_consensus.cc:1226] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:06.996416 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:06.996955 668 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:06.982264 5193 raft_consensus.cc:2781] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:07.001298 501 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:06.987440 5252 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:44:06.985549 4786 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:07.020714 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.023156 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.026341 664 raft_consensus.cc:2396] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260806 03:44:07.012162 403 catalog_manager.cc:5495] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:06.988045 5262 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260806 03:44:07.002288 4119 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:07.033567 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:07.034252 768 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:07.020146 4405 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:44:06.998232 4909 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
I20260806 03:44:07.042502 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.044083 727 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:07.027130 5193 raft_consensus.cc:686] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:07.045374 667 raft_consensus.cc:2454] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:07.064419 1662 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:07.065392 403 catalog_manager.cc:5495] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 7, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 7 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:07.040184 4119 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.089756 1671 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:44:07.089579 4786 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.043335 4405 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:07.097258 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 122067us
I20260806 03:44:07.096390 403 catalog_manager.cc:5495] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 6, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67). New cstate: current_term: 6 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.046722 5125 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:07.099100 727 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:07.100045 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.101464 4405 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:44:07.119310 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.119599 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:07.130719 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 156662us
I20260806 03:44:07.142513 403 catalog_manager.cc:5495] T 3d8ac3ebccff4054953475f839218e45 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:07.113006 4909 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:07.143126 768 raft_consensus.cc:2454] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
W20260806 03:44:07.146544 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 47573.3us
I20260806 03:44:07.112802 5125 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 03:44:07.166842 667 raft_consensus.cc:1226] T aa826ec82b0a402580f530e7eb832852 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 4. Current term is 5. Ops: []
I20260806 03:44:07.162971 1671 raft_consensus.cc:1226] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:07.150259 403 catalog_manager.cc:5495] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.172338 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.165539 775 raft_consensus.cc:1261] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:44:07.162226 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:07.174791 1658 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:44:07.134204 5253 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:07.135025 4119 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:07.165928 400 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35370 (request call id 319) took 32554 ms (client timeout 15000 ms). Trace:
I20260806 03:44:07.180668 769 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:07.198020 727 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:07.170071 4909 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 03:44:07.241510 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 86680.4us
I20260806 03:44:07.180748 5253 raft_consensus.cc:482] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:44:07.303642 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 145720us
I20260806 03:44:07.305723 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:07.307178 5243 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:07.308399 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.266937 403 catalog_manager.cc:5495] T 20fcbc41710a42cd9ee5dacc5d9577fb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 6, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to ada5ef6656cb4420baf77a4b4342909e (127.30.138.65). New cstate: current_term: 6 leader_uuid: "ada5ef6656cb4420baf77a4b4342909e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.170243 5255 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260806 03:44:07.309449 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.311683 674 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:07.318581 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.311288 771 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:44:07.305450 5264 consensus_queue.cc:260] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.316169 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.311472 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.330652 674 raft_consensus.cc:2379] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:07.330724 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.331568 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.149505 5193 consensus_queue.cc:237] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.268527 5253 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:07.342180 5191 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:07.342571 5312 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:07.347067 5125 raft_consensus.cc:3032] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:07.347723 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.329473 4786 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:07.391887 773 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:07.327885 5262 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
W20260806 03:44:07.405108 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 72007.9us
I20260806 03:44:07.406179 1624 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:07.398865 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.353617 403 catalog_manager.cc:5495] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 3 to 5, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.365612 5191 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:07.407439 1624 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:07.417685 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.375491 5125 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:07.421800 5191 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:07.445132 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 266361us
I20260806 03:44:07.446784 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:07.446959 727 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:07.448025 5255 raft_consensus.cc:2741] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260806 03:44:07.419983 4909 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:07.473441 562 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.521s user 0.012s sys 0.000s
I20260806 03:44:07.473986 562 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:07.475715 403 catalog_manager.cc:5495] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 5 to 6, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
W20260806 03:44:07.184031 400 rpcz_store.cc:269] 0806 03:43:34.485970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:43:34.531539 (+ 45569us) service_pool.cc:227] Handling call
0806 03:44:07.009296 (+32477757us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:44:00.740759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:44:00.928099 (+187340us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:44:00.940939 (+ 12840us) write_op.cc:435] Acquired schema lock
0806 03:44:00.940968 (+ 29us) tablet.cc:660] Decoding operations
0806 03:44:01.139182 (+198214us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:44:01.152523 (+ 13341us) write_op.cc:641] Partition lock acquired for write op
0806 03:44:01.158434 (+ 5911us) tablet.cc:683] Acquiring locks for 6 operations
0806 03:44:01.158984 (+ 550us) tablet.cc:699] Row locks acquired
0806 03:44:01.159000 (+ 16us) write_op.cc:260] PREPARE: finished
0806 03:44:01.159152 (+ 152us) write_op.cc:270] Start()
0806 03:44:01.159259 (+ 107us) write_op.cc:276] Timestamp: P: 1785987841159103 usec, L: 0
0806 03:44:01.159278 (+ 19us) op_driver.cc:331] REPLICATION: starting
0806 03:44:01.232728 (+ 73450us) log.cc:842] Serialized 2467 byte log entry
0806 03:44:04.773349 (+3540621us) op_driver.cc:444] REPLICATION: finished
0806 03:44:05.859499 (+1086150us) write_op.cc:301] APPLY: starting
0806 03:44:05.879511 (+ 20012us) tablet.cc:1365] starting BulkCheckPresence
0806 03:44:05.893875 (+ 14364us) tablet.cc:1368] finished BulkCheckPresence
0806 03:44:05.894020 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:44:05.918421 (+ 24401us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:44:05.918791 (+ 370us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0806 03:44:05.920233 (+ 1442us) write_op.cc:312] APPLY: finished
0806 03:44:05.923394 (+ 3161us) log.cc:842] Serialized 53 byte log entry
0806 03:44:05.978133 (+ 54739us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:44:05.999931 (+ 21798us) write_op.cc:454] Released schema lock
0806 03:44:06.006494 (+ 6563us) write_op.cc:341] FINISH: Updating metrics
0806 03:44:06.038554 (+ 32060us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:44:06.040891 (+ 2337us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":292496,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":1052693,"apply.run_cpu_time_us":49976,"apply.run_wall_time_us":258557,"num_ops":6,"prepare.queue_time_us":465470,"prepare.run_cpu_time_us":30924,"prepare.run_wall_time_us":897222,"raft.queue_time_us":1392206,"raft.run_cpu_time_us":99459,"raft.run_wall_time_us":994604,"replication_time_us":3613848,"thread_start_us":2961051,"threads_started":5,"wal-append.queue_time_us":937783,"wal-append.run_cpu_time_us":57275,"wal-append.run_wall_time_us":4650365}]]}
I20260806 03:44:07.423831 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:07.408661 665 raft_consensus.cc:2454] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:07.333410 4119 consensus_queue.cc:1046] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.070s
I20260806 03:44:07.396509 4786 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:07.493664 403 catalog_manager.cc:5495] T 7a9afce5efa4491797ebed2aab67532f P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 5, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:07.450423 5147 consensus_queue.cc:1035] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.024s
W20260806 03:44:07.498948 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 75213.9us
I20260806 03:44:07.503865 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:44:07.504344 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 112353us
I20260806 03:44:07.478559 5305 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader election won for term 9
W20260806 03:44:07.508258 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 200112us
I20260806 03:44:07.517266 501 leader_election.cc:302] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:07.519651 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.522689 31273 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:44:07.503706 5265 raft_consensus.cc:2741] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 03:44:07.520222 615 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:07.532801 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.538687 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 63871.8us
I20260806 03:44:07.543021 31273 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:44:07.543933 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:44:07.483654 5308 consensus_queue.cc:1046] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260806 03:44:07.547827 5308 consensus_queue.cc:1046] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:44:07.549176 5308 raft_consensus.cc:3032] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:07.549608 5308 raft_consensus.cc:729] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:07.520364 5198 raft_consensus.cc:3032] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 03:44:07.503988 4909 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:07.555910 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.524242 5305 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:07.550765 5262 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260806 03:44:07.576635 5310 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260806 03:44:07.576828 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.577833 5317 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 03:44:07.577893 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.518730 5333 raft_consensus.cc:2781] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:07.575127 5312 consensus_queue.cc:1035] T c9369dafb3f342e7a15d44086a377b55 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:44:07.580484 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:07.596815 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.597754 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.598179 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:07.598508 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.599021 5305 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.557318 5198 raft_consensus.cc:729] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:07.605541 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.555184 4963 consensus_meta.cc:287] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.648s user 0.008s sys 0.004s
I20260806 03:44:07.602473 403 catalog_manager.cc:5495] T e8a4c59f28ea45f6998c3f7f0b7dda53 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:44:07.587630 5316 log.cc:924] Time spent T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260806 03:44:07.615068 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:07.615281 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.612219 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 95827.8us
I20260806 03:44:07.617367 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:07.619398 5326 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:07.557752 5308 consensus_queue.cc:260] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.580513 5262 raft_consensus.cc:482] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:07.620666 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.620877 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.557797 5265 raft_consensus.cc:2726] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:07.625828 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.665890 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:07.683429 767 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:07.666414 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.665094 674 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:07.686190 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.625676 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.630664 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.616215 664 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:07.695938 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:07.696753 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 289395us
I20260806 03:44:07.694454 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.591236 4953 raft_consensus.cc:3032] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:07.700307 665 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:07.707964 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.704866 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.696666 768 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:07.695837 563 raft_consensus.cc:2362] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6. Re-sending same reply.
I20260806 03:44:07.727664 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.697041 403 catalog_manager.cc:5495] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 5, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.536335 5125 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.625497 5308 raft_consensus.cc:3037] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:07.732476 769 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:07.735615 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:07.761068 772 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:07.736408 544 raft_consensus.cc:1261] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:44:07.738598 775 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:07.616909 5333 raft_consensus.cc:686] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:07.745872 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.746341 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.747340 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.748831 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 57799.2us
I20260806 03:44:07.759037 403 catalog_manager.cc:5495] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 5, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 5 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:07.770131 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.698390 5265 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:07.771378 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.771747 1658 raft_consensus.cc:1226] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:07.702911 5262 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:07.767563 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.768446 562 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:07.774741 403 catalog_manager.cc:5495] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:07.829200 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.830412 659 raft_consensus.cc:1261] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:44:07.844717 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.845016 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:07.803462 4786 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:07.806257 5338 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:07.858909 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fcbc41710a42cd9ee5dacc5d9577fb" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.859063 5338 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:07.851632 767 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 9.
I20260806 03:44:07.707410 4953 raft_consensus.cc:729] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:07.809846 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:07.620029 5326 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.759478 5198 consensus_queue.cc:260] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.802608 5318 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:07.790975 5333 consensus_queue.cc:237] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.825531 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.826249 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.827414 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 60588.6us
W20260806 03:44:07.767041 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 166225us
I20260806 03:44:07.836843 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.852571 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:07.852989 501 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:07.853636 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.854230 563 raft_consensus.cc:2396] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 03:44:07.828253 5191 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:07.855940 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:07.856446 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:07.826303 5307 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.378s user 0.001s sys 0.002s
I20260806 03:44:07.856695 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.857105 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.857885 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.857259 403 catalog_manager.cc:5495] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 4 to 8, leader changed from d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66) to 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68). New cstate: current_term: 8 leader_uuid: "1914b652c9fb49b694d71a5f5f5752ee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:44:07.822585 5264 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:44:07.830662 5296 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.730s user 0.002s sys 0.001s
I20260806 03:44:07.857277 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1458) took 1211 ms. Trace:
W20260806 03:44:07.810333 5252 consensus_meta.cc:287] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.780s user 0.011s sys 0.002s
I20260806 03:44:07.965075 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.007153 561 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:07.839938 5265 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:07.997452 542 rpcz_store.cc:276] 0806 03:44:06.646225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:06.930047 (+283822us) service_pool.cc:227] Handling call
0806 03:44:06.930440 (+ 393us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:06.930823 (+ 383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:06.930856 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:06.931444 (+ 588us) log.cc:842] Serialized 26 byte log entry
0806 03:44:07.096948 (+165504us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:44:07.117949 (+ 21001us) log.cc:842] Serialized 14 byte log entry
0806 03:44:07.118272 (+ 323us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:07.118401 (+ 129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:07.811220 (+692819us) spinlock_profiling.cc:243] Waited 5.03 ms on lock 0x7b180042d1e0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321c2968b1 00007f321c2a3661 00007f32133f7638 00007f32133fab11 00007f32133faac9 00007f32133faa59 00007f32133faa21 00007f32133f9d1d 00007f321c65b204 00007f321c65b039 00007f32133d9dc6
0806 03:44:07.844415 (+ 33195us) raft_consensus.cc:1671] finished
0806 03:44:07.857038 (+ 12623us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:07.857192 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8848,"prepare.queue_time_us":18310,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2399,"raft.queue_time_us":1357,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":16125,"spinlock_wait_cycles":11571968,"thread_start_us":165042,"threads_started":1,"wal-append.queue_time_us":214543}
I20260806 03:44:07.965423 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:07.997956 558 raft_consensus.cc:2454] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:07.997365 548 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:07.863726 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:07.974289 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 164503us
I20260806 03:44:08.016764 769 raft_consensus.cc:2379] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:08.016855 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:07.813295 4911 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:07.927017 4405 consensus_meta.cc:287] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.793s user 0.012s sys 0.000s
I20260806 03:44:07.923893 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.044915 544 raft_consensus.cc:2454] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
W20260806 03:44:08.020855 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 167960us
I20260806 03:44:07.868767 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:08.046514 767 raft_consensus.cc:2379] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 6: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:44:07.971640 5253 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.596s user 0.011s sys 0.000s
I20260806 03:44:08.009654 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.024730 403 catalog_manager.cc:5495] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee reported cstate change: term changed from 2 to 5, leader changed from 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } } }
I20260806 03:44:08.023741 5265 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:08.048934 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.049672 562 raft_consensus.cc:3037] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:44:08.027508 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8ac3ebccff4054953475f839218e45" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:08.051113 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.051924 563 raft_consensus.cc:2354] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7.
I20260806 03:44:08.052781 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.053148 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.053930 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:07.966059 4909 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:08.054859 4909 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:08.055446 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.056272 667 raft_consensus.cc:2396] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:08.057179 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.058053 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:08.058565 5253 raft_consensus.cc:504] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:08.059031 5026 consensus_meta.cc:287] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.053s user 0.003s sys 0.010s
I20260806 03:44:08.059470 5026 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:08.060294 769 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:08.060704 5253 leader_election.cc:288] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:08.054713 563 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:08.061287 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.061573 5026 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:08.062409 661 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 10: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:08.062613 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.063474 542 raft_consensus.cc:2396] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 03:44:08.063599 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.061368 5253 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:08.062186 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:07.967237 5338 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:08.066468 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:08.054659 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.067934 558 raft_consensus.cc:2379] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:08.068769 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.069283 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.054276 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.009824 546 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:08.074339 768 raft_consensus.cc:1261] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:44:08.054239 775 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:44:08.076509 659 raft_consensus.cc:2396] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:08.011026 5312 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260806 03:44:08.084223 1622 leader_election.cc:334] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.594s (SENT)
I20260806 03:44:08.077730 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.073290 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.080010 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.080740 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.088157 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.088070 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.089211 659 raft_consensus.cc:2396] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260806 03:44:08.090890 674 raft_consensus.cc:2379] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:08.091586 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.091784 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.092407 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.078671 562 raft_consensus.cc:2454] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5.
I20260806 03:44:08.106771 542 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:08.108029 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.093205 548 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 11.
W20260806 03:44:08.108970 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 405387us
I20260806 03:44:08.133801 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.134635 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.135090 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:07.961545 4953 consensus_queue.cc:260] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:08.067695 4405 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.006654 5252 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.078212 4911 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.455817 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:08.456072 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.089075 5191 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:08.150149 544 raft_consensus.cc:1261] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Refusing update from remote peer 0374a84d5e9f4e3c8d8287e2382379c8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:44:08.065774 5318 consensus_queue.cc:1225] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260806 03:44:08.772167 767 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:44:08.156018 770 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 675) took 1219 ms. Trace:
I20260806 03:44:08.469318 501 leader_election.cc:302] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:08.474494 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.137872 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:08.773548 770 rpcz_store.cc:276] 0806 03:44:06.936318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:06.960012 (+ 23694us) service_pool.cc:227] Handling call
0806 03:44:06.960431 (+ 419us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:06.961822 (+ 1391us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:44:06.961861 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:06.976090 (+ 14229us) log.cc:842] Serialized 26 byte log entry
0806 03:44:07.177804 (+201714us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:44:07.177838 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:07.177906 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:07.825486 (+647580us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:44:08.147880 (+322394us) raft_consensus.cc:1671] finished
0806 03:44:08.147909 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.155909 (+ 8000us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7496960,"thread_start_us":199006,"threads_started":1,"wal-append.queue_time_us":469378}
W20260806 03:44:08.775072 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 305879us
I20260806 03:44:08.136129 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.775748 563 raft_consensus.cc:2454] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:08.775808 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:08.769726 561 raft_consensus.cc:2362] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8. Re-sending same reply.
I20260806 03:44:08.771376 776 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:44:08.776947 501 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:08.767424 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2462) took 1375 ms. Trace:
I20260806 03:44:08.463332 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1814) took 1138 ms. Trace:
I20260806 03:44:08.767554 1663 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260806 03:44:08.768461 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 751707us
I20260806 03:44:08.777910 661 rpcz_store.cc:276] 0806 03:44:07.309101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.309217 (+ 116us) service_pool.cc:227] Handling call
0806 03:44:08.447684 (+1138467us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54000b2554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:08.447903 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1712421120}
I20260806 03:44:08.780004 661 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 03:44:08.780510 727 connection.cc:232] RPC call timeout handler was delayed by -1.0825862009078264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
I20260806 03:44:08.780745 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 441) took 1335 ms. Trace:
I20260806 03:44:08.774418 664 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:44:08.781231 661 rpcz_store.cc:276] 0806 03:44:07.445747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.779319 (+1333572us) service_pool.cc:227] Handling call
0806 03:44:08.779643 (+ 324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.780474 (+ 831us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.780677 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.782112 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2470) took 1231 ms. Trace:
I20260806 03:44:08.782537 664 rpcz_store.cc:276] 0806 03:44:07.551047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.474055 (+923008us) service_pool.cc:227] Handling call
0806 03:44:08.782048 (+307993us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.783155 661 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260806 03:44:08.666862 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 339740us
W20260806 03:44:08.363381 1662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.009s sys 0.006s
I20260806 03:44:08.304859 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:08.504339 564 raft_consensus.cc:2354] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 4. Current term is 5.
I20260806 03:44:08.148455 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:08.150229 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.304996 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.597018 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:08.788834 674 raft_consensus.cc:3037] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:08.789042 766 raft_consensus.cc:2396] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:44:08.784963 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 443) took 1337 ms. Trace:
I20260806 03:44:08.777724 668 rpcz_store.cc:276] 0806 03:44:07.391870 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.455791 (+1063921us) service_pool.cc:227] Handling call
0806 03:44:08.767358 (+311567us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:08.768985 1622 leader_election.cc:334] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.652s (SENT)
I20260806 03:44:08.185206 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:08.789537 661 rpcz_store.cc:276] 0806 03:44:07.446354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.782404 (+1336050us) service_pool.cc:227] Handling call
0806 03:44:08.782789 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.783687 (+ 898us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.783982 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:08.784519 1662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s user 0.022s sys 0.027s
I20260806 03:44:08.784143 616 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:08.147217 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.775245 772 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
W20260806 03:44:08.645206 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 46667.8us
W20260806 03:44:08.781026 727 connection.cc:232] RPC call timeout handler was delayed by -1.0825862009078264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
I20260806 03:44:08.765946 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.791796 1670 raft_consensus.cc:2354] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:08.790932 1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 552) took 1085 ms. Trace:
I20260806 03:44:08.792519 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 454) took 1164 ms. Trace:
W20260806 03:44:08.605585 5351 log.cc:924] Time spent T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
W20260806 03:44:08.605859 5262 consensus_meta.cc:287] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.747s user 0.004s sys 0.007s
W20260806 03:44:08.457517 5314 log.cc:924] Time spent T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.911s user 0.010s sys 0.001s
I20260806 03:44:08.432828 4934 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:08.791533 727 connection.cc:232] RPC call timeout handler was delayed by -1.066386200953275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
W20260806 03:44:08.612911 5352 log.cc:924] Time spent T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.285s user 0.002s sys 0.000s
I20260806 03:44:08.793015 542 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:08.830790 663 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:08.794102 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:08.546581 5264 consensus_meta.cc:287] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.547s user 0.005s sys 0.005s
W20260806 03:44:08.609133 5353 log.cc:924] Time spent T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260806 03:44:08.796252 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.834257 547 raft_consensus.cc:2396] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 03:44:08.796895 766 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2416) took 1045 ms. Trace:
I20260806 03:44:08.797277 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.835668 544 raft_consensus.cc:2396] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260806 03:44:08.830417 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2467) took 1291 ms. Trace:
I20260806 03:44:08.798725 669 raft_consensus.cc:2379] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:08.800235 546 raft_consensus.cc:1261] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:44:08.836582 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 456) took 1019 ms. Trace:
I20260806 03:44:08.800981 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.792325 1662 rpcz_store.cc:276] 0806 03:44:07.705204 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.705520 (+ 316us) service_pool.cc:227] Handling call
0806 03:44:07.772277 (+ 66757us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:07.830756 (+ 58479us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:07.830790 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:07.831208 (+ 418us) log.cc:842] Serialized 26 byte log entry
0806 03:44:08.790582 (+959374us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:08.790621 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.790666 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:08.790741 (+ 75us) raft_consensus.cc:1671] finished
0806 03:44:08.790754 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.790865 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22357,"raft.queue_time_us":22573,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":959001,"threads_started":1,"wal-append.queue_time_us":734507}
W20260806 03:44:08.187933 5206 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.004s sys 0.013s
W20260806 03:44:08.801985 1622 leader_election.cc:334] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.827s (SENT)
W20260806 03:44:08.841730 1622 leader_election.cc:334] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.832s (SENT)
I20260806 03:44:08.802503 616 leader_election.cc:302] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:08.803553 5287 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:08.804014 5355 raft_consensus.cc:2781] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:08.432010 5323 raft_consensus.cc:482] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.806273 5198 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:44:08.808175 728 leader_election.cc:398] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:08.771173 5243 consensus_queue.cc:1046] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260806 03:44:08.843256 671 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:08.479693 5265 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.604882 5342 consensus_queue.cc:1046] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.206s
I20260806 03:44:08.422322 5193 consensus_queue.cc:1046] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.389s
I20260806 03:44:08.794114 5191 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:08.813859 4405 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:08.096840 5332 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.699s
I20260806 03:44:08.819048 5362 consensus_queue.cc:1046] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:44:08.820233 5354 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:08.875226 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.821465 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.573319 5326 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:08.787416 614 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.815s (SENT) [suppressed 35 similar messages]
I20260806 03:44:08.666605 5312 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.830224 776 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:08.831013 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.791268 772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2418) took 1022 ms. Trace:
I20260806 03:44:08.831715 558 raft_consensus.cc:2454] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:08.832417 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 460) took 1003 ms. Trace:
I20260806 03:44:08.890899 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.891345 542 rpcz_store.cc:276] 0806 03:44:07.829045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.107504 (+278459us) service_pool.cc:227] Handling call
0806 03:44:08.832329 (+724825us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.793057 659 rpcz_store.cc:276] 0806 03:44:07.627876 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.117178 (+489302us) service_pool.cc:227] Handling call
0806 03:44:08.792460 (+675282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.836177 766 rpcz_store.cc:276] 0806 03:44:07.751197 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.784134 (+ 32937us) service_pool.cc:227] Handling call
0806 03:44:08.304672 (+520538us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:44:08.796830 (+492158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7540480}
I20260806 03:44:08.839003 563 raft_consensus.cc:2354] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 4. Current term is 5.
I20260806 03:44:08.892366 562 raft_consensus.cc:2354] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7.
I20260806 03:44:08.836122 664 rpcz_store.cc:276] 0806 03:44:07.508485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.783243 (+1274758us) service_pool.cc:227] Handling call
0806 03:44:08.783518 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.799210 (+ 15692us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.799536 (+ 326us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:08.799605 (+ 69us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:44:08.799661 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.799745 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.799884 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11597,"raft.queue_time_us":12582,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5221}
I20260806 03:44:08.837126 546 rpcz_store.cc:276] 0806 03:44:07.817629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.132306 (+314677us) service_pool.cc:227] Handling call
0806 03:44:08.798179 (+665873us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.836328 (+ 38149us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.836530 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.789553 548 raft_consensus.cc:2454] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:08.842902 616 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:08.845326 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35214 (request call id 1948) took 1243 ms. Trace:
I20260806 03:44:08.894137 564 rpcz_store.cc:276] 0806 03:44:07.597352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.597493 (+ 141us) service_pool.cc:227] Handling call
0806 03:44:08.840592 (+1243099us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540031ecd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:08.840719 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2071169408}
I20260806 03:44:08.835781 4934 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:08.793310 5253 raft_consensus.cc:2726] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:08.844556 5264 raft_consensus.cc:2726] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 03:44:08.850678 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 71173.9us
I20260806 03:44:08.895788 564 raft_consensus.cc:1261] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 03:44:08.868076 671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1815) took 1557 ms (client timeout 1941 ms). Trace:
W20260806 03:44:08.842844 5206 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.178s user 0.009s sys 0.045s
W20260806 03:44:08.896663 671 rpcz_store.cc:269] 0806 03:44:07.310647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.310818 (+ 171us) service_pool.cc:227] Handling call
0806 03:44:08.867792 (+1556974us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540016e9d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:08.867998 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3374486784}
I20260806 03:44:08.851014 5355 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:08.897519 5355 raft_consensus.cc:3037] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:08.850893 5287 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:08.863487 669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48530 (request call id 2450) took 1965 ms (client timeout 1884 ms). Trace:
I20260806 03:44:08.871121 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:08.871536 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 94690.6us
I20260806 03:44:08.908839 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:08.910503 501 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:08.865638 5026 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:08.820724 1671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60272 (request call id 553) took 1114 ms. Trace:
I20260806 03:44:08.872090 5191 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:08.850301 728 leader_election.cc:401] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:44:08.900213 1664 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:44:08.890390 772 rpcz_store.cc:276] 0806 03:44:07.768529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.792531 (+ 24002us) service_pool.cc:227] Handling call
0806 03:44:08.137689 (+345158us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:44:08.791214 (+653525us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6425984}
I20260806 03:44:08.901408 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2414) took 1162 ms. Trace:
I20260806 03:44:08.921381 776 rpcz_store.cc:276] 0806 03:44:07.728294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.746553 (+ 18259us) service_pool.cc:227] Handling call
0806 03:44:08.134899 (+388346us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800004298. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:44:08.890200 (+755301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6410112}
W20260806 03:44:08.870041 1622 leader_election.cc:334] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.155s (SENT)
I20260806 03:44:08.893685 1657 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60246 (request call id 692) took 1573 ms. Trace:
I20260806 03:44:08.922833 1657 rpcz_store.cc:276] 0806 03:44:07.320083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.320261 (+ 178us) service_pool.cc:227] Handling call
0806 03:44:07.336560 (+ 16299us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:07.508879 (+172319us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:07.508909 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:07.521270 (+ 12361us) log.cc:842] Serialized 26 byte log entry
0806 03:44:07.998681 (+477411us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:44:07.998721 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:07.998785 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:08.893395 (+894610us) raft_consensus.cc:1671] finished
0806 03:44:08.893423 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.893610 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61444,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":21467,"thread_start_us":619876,"threads_started":2,"wal-append.queue_time_us":573298}
I20260806 03:44:08.845451 5360 raft_consensus.cc:482] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.897879 564 raft_consensus.cc:1261] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 03:44:08.904392 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:08.934094 542 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:08.905557 5326 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.906104 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:08.905951 669 rpcz_store.cc:269] 0806 03:44:06.883554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:06.883754 (+ 200us) service_pool.cc:227] Handling call
0806 03:44:08.848426 (+1964672us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b54002ce554. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:08.848576 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4354548992}
I20260806 03:44:08.907626 674 raft_consensus.cc:2454] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:08.939836 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 453) took 1312 ms. Trace:
I20260806 03:44:08.838117 5361 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.936345 5326 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.917392 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:08.945667 5361 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:08.946674 558 raft_consensus.cc:2379] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:08.947654 5326 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:08.947798 5368 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:08.948041 5361 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:08.948599 5361 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823) with higher term. Message: Invalid argument: T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
W20260806 03:44:08.949635 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 106813us
I20260806 03:44:08.950057 542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 458) took 1131 ms. Trace:
I20260806 03:44:08.940408 664 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:44:08.936209 5026 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:08.951642 562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 459) took 1123 ms. Trace:
I20260806 03:44:08.958297 562 rpcz_store.cc:276] 0806 03:44:07.828615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.905778 (+1077163us) service_pool.cc:227] Handling call
0806 03:44:08.920131 (+ 14353us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.935221 (+ 15090us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.935267 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:08.951339 (+ 16072us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:08.951402 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.951481 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.951587 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.892710 561 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:08.940327 674 rpcz_store.cc:276] 0806 03:44:07.627579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.121020 (+493441us) service_pool.cc:227] Handling call
0806 03:44:08.939779 (+818759us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.831053 5262 raft_consensus.cc:504] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:08.950796 542 rpcz_store.cc:276] 0806 03:44:07.818293 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.905725 (+1087432us) service_pool.cc:227] Handling call
0806 03:44:08.929671 (+ 23946us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.949786 (+ 20115us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.949998 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:08.951817 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 164411us
W20260806 03:44:08.952464 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 82463.4us
W20260806 03:44:08.961783 1622 leader_election.cc:334] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.182s (SENT)
I20260806 03:44:08.954708 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 459) took 1325 ms. Trace:
I20260806 03:44:08.915771 1671 rpcz_store.cc:276] 0806 03:44:07.705893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.706007 (+ 114us) service_pool.cc:227] Handling call
0806 03:44:07.771077 (+ 65070us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:07.827592 (+ 56515us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:07.827625 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:07.828149 (+ 524us) log.cc:842] Serialized 26 byte log entry
0806 03:44:08.304672 (+476523us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:08.304714 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.304793 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:08.820490 (+515697us) raft_consensus.cc:1671] finished
0806 03:44:08.820511 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.820668 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56061,"raft.queue_time_us":57347,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":476110,"threads_started":1,"wal-append.queue_time_us":990394}
I20260806 03:44:08.912000 5253 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:08.963657 664 rpcz_store.cc:276] 0806 03:44:07.629423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.953605 (+1324182us) service_pool.cc:227] Handling call
0806 03:44:08.954027 (+ 422us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.954268 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.954305 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:08.954361 (+ 56us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:08.954432 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.954501 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.954634 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:44:08.966323 1622 leader_election.cc:334] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.182s (SENT)
I20260806 03:44:08.968073 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 458) took 1339 ms. Trace:
I20260806 03:44:08.968622 669 rpcz_store.cc:276] 0806 03:44:07.629118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.939813 (+1310695us) service_pool.cc:227] Handling call
0806 03:44:08.967408 (+ 27595us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:08.967655 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.967692 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:44:08.967749 (+ 57us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:44:08.967807 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:08.967879 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:08.967997 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:08.970561 563 raft_consensus.cc:1261] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:44:08.932269 5191 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:08.963697 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 164565us
W20260806 03:44:08.963166 5217 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.761s user 0.013s sys 0.036s
I20260806 03:44:08.967069 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:08.902316 4405 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260806 03:44:08.979990 5125 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:08.978560 664 raft_consensus.cc:1261] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 03:44:09.001596 664 pending_rounds.cc:76] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Aborting all ops after (but not including) 6
I20260806 03:44:09.001854 664 pending_rounds.cc:97] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260806 03:44:09.001988 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 77671.7us
I20260806 03:44:08.991772 665 raft_consensus.cc:2354] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 9. Current term is 10.
I20260806 03:44:09.002828 501 leader_election.cc:302] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:08.902522 5312 raft_consensus.cc:504] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:08.993525 5217 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.014s sys 0.036s
I20260806 03:44:08.996318 5262 leader_election.cc:288] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:08.965942 5264 raft_consensus.cc:2781] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:09.002131 664 raft_consensus.cc:2866] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:44:08.866417 5323 raft_consensus.cc:504] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:09.003487 663 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
W20260806 03:44:09.027560 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 45063.4us
I20260806 03:44:09.037278 664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:36546 (request call id 550) took 1538 ms. Trace:
I20260806 03:44:09.004382 5252 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:09.013846 5343 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:09.037791 664 rpcz_store.cc:276] 0806 03:44:07.498952 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:09.036017 (+1537065us) service_pool.cc:227] Handling call
0806 03:44:09.037196 (+ 1179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:09.020109 5362 raft_consensus.cc:2726] T 3d8ac3ebccff4054953475f839218e45 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:44:09.075418 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 57477.2us
I20260806 03:44:08.999944 5253 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:09.078902 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 79983.1us
I20260806 03:44:08.962293 5360 raft_consensus.cc:504] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:09.043352 5264 raft_consensus.cc:686] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:09.042673 5369 consensus_queue.cc:1035] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260806 03:44:09.084669 501 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:09.089200 727 leader_election.cc:334] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.742s (SENT)
I20260806 03:44:09.089658 727 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:09.035781 5206 consensus_queue.cc:1046] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.053s
I20260806 03:44:09.057981 5329 consensus_queue.cc:1035] T fdbf67cdfa6f4f2785238e161fd2c4c8 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260806 03:44:09.155124 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:09.296388 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:09.163129 663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1821) took 1823 ms (client timeout 2218 ms). Trace:
I20260806 03:44:09.164242 674 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:09.301120 1662 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:09.283738 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:09.172832 616 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:09.303592 772 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 9.
I20260806 03:44:09.305809 1624 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:09.136348 501 leader_election.cc:302] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:09.307114 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 170853us
I20260806 03:44:09.170060 5372 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:09.311565 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
W20260806 03:44:09.252081 665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36554 (request call id 1824) took 1810 ms (client timeout 2205 ms). Trace:
I20260806 03:44:09.082571 4934 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:09.301491 663 rpcz_store.cc:269] 0806 03:44:07.340240 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.340376 (+ 136us) service_pool.cc:227] Handling call
0806 03:44:07.823383 (+483007us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bf851 00007f32133bf702 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:44:07.852365 (+ 28982us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b58001f2098. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f32133bfb7c 00007f32133bf73b 00007f321c2afb09 00007f321b1eb730 00007f321b2b179b 00007f321b2b1821 00007f321b2611b0 00007f321b260a94 00007f3214948c04 00007f3214948b92
0806 03:44:09.162852 (+1310487us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b540019afd4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:09.163050 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957120512}
I20260806 03:44:09.088425 5265 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:09.327387 665 rpcz_store.cc:269] 0806 03:44:07.442201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:07.745541 (+303340us) service_pool.cc:227] Handling call
0806 03:44:09.251828 (+1506287us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002ce054. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:09.251999 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2792653696}
I20260806 03:44:09.163867 4953 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:09.365116 4405 raft_consensus.cc:2781] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:44:09.087837 4911 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:09.796711 5308 raft_consensus.cc:3032] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:09.422129 5323 leader_election.cc:288] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.103122 5308 raft_consensus.cc:729] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:44:09.686363 5387 log.cc:924] Time spent T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260806 03:44:09.705379 5356 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:10.104236 5308 consensus_queue.cc:260] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:09.171518 5264 consensus_queue.cc:237] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:09.297592 5026 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.105834 5308 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:09.411375 5360 leader_election.cc:288] T 864f4306a7964687b312ccf31fe6797a P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:09.363763 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:09.338694 5191 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:09.737087 5253 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:10.109323 5191 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:09.353821 5333 raft_consensus.cc:2781] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260806 03:44:09.512553 5355 consensus_meta.cc:287] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.615s user 0.004s sys 0.007s
I20260806 03:44:10.110064 5333 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:09.251382 5354 raft_consensus.cc:3032] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:10.110633 5333 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:09.388548 766 pending_rounds.cc:76] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting all ops after (but not including) 2
I20260806 03:44:10.111455 766 pending_rounds.cc:97] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:44:10.111784 766 raft_consensus.cc:2866] T 1c7b59fcc4bf4cafad47c9aed88c416e P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:44:09.535073 1662 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 10.
I20260806 03:44:09.165218 4786 raft_consensus.cc:3032] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:09.753450 5318 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:10.110920 5354 raft_consensus.cc:729] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:10.119262 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:10.030524 5198 consensus_meta.cc:287] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 1.164s user 0.014s sys 0.002s
W20260806 03:44:09.861133 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 52369.8us
I20260806 03:44:09.090662 5326 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:09.337257 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.138317 772 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:09.768301 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.121337 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:10.140420 768 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.107039 767 raft_consensus.cc:3037] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:44:10.123553 561 raft_consensus.cc:1261] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:44:09.547832 5312 leader_election.cc:288] T 2c5936af1dcf436eb5f06e8fc0765e83 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.125304 1657 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:44:10.101275 546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58044 (request call id 463) took 2088 ms. Trace:
W20260806 03:44:09.741297 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 660704us
I20260806 03:44:09.343602 770 raft_consensus.cc:1261] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:44:10.144671 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7b59fcc4bf4cafad47c9aed88c416e" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:10.143602 546 rpcz_store.cc:276] 0806 03:44:08.012829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.905492 (+892663us) service_pool.cc:227] Handling call
0806 03:44:08.928767 (+ 23275us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:08.929173 (+ 406us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.929212 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:08.938666 (+ 9454us) log.cc:842] Serialized 26 byte log entry
0806 03:44:09.086976 (+148310us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:09.364472 (+277496us) log.cc:842] Serialized 14 byte log entry
0806 03:44:10.100990 (+736518us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:10.101058 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:10.101098 (+ 40us) raft_consensus.cc:1671] finished
0806 03:44:10.101111 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:10.101218 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":328857,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":769922,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"thread_start_us":317971,"threads_started":2,"wal-append.queue_time_us":407804}
I20260806 03:44:10.098794 771 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:09.273787 5287 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:10.148658 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:09.257486 5368 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:10.133590 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:09.330853 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 282750us
W20260806 03:44:09.691287 5370 log.cc:924] Time spent T 00000000000000000000000000000000 P eed80bed2e2c4b49a8999c7f5e713c34: Append to log took a long time: real 0.399s user 0.003s sys 0.000s
W20260806 03:44:09.691416 5375 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260806 03:44:10.099705 775 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 03:44:09.731076 5125 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.717s user 0.010s sys 0.002s
I20260806 03:44:10.114841 5394 raft_consensus.cc:3032] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:10.138572 615 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:10.149609 770 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.118983 498 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.147802 5206 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.105141 5356 raft_consensus.cc:686] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:10.110464 4405 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.181327 4405 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:10.181768 5356 consensus_queue.cc:237] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.168737 5323 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.185387 5323 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.142182 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:10.151038 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.187086 773 raft_consensus.cc:2379] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:10.121634 4786 raft_consensus.cc:729] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:10.187614 5323 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.188258 4786 consensus_queue.cc:260] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.155117 5354 consensus_queue.cc:260] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.132619 5318 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:10.190112 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 71248.4us
I20260806 03:44:10.168175 5191 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.023s
I20260806 03:44:10.191272 498 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:10.137792 5338 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.163404 5265 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:10.165673 768 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:10.130162 5373 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.169296 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.180476 5206 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.190163 5354 raft_consensus.cc:3037] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:10.151686 5368 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:10.164026 5396 log.cc:924] Time spent T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.893s user 0.013s sys 0.004s
I20260806 03:44:10.179296 776 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:10.194149 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.193996 767 raft_consensus.cc:2454] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 5.
I20260806 03:44:10.197865 501 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:10.189456 4786 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.150098 5326 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.211300 674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2483) took 2389 ms. Trace:
I20260806 03:44:10.211830 674 rpcz_store.cc:276] 0806 03:44:07.821882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.959738 (+1137856us) service_pool.cc:227] Handling call
0806 03:44:09.164023 (+204285us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:09.301630 (+137607us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:10.211117 (+909487us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540019afd0. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ea09 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:10.211244 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468761984}
I20260806 03:44:10.134335 5243 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260806 03:44:10.110460 5355 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.171120 5394 raft_consensus.cc:729] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
W20260806 03:44:10.200170 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 56244.3us
I20260806 03:44:10.221877 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:10.222187 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.209757 501 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:10.222807 768 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:10.222879 661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 455) took 2594 ms. Trace:
I20260806 03:44:10.204669 5368 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:10.216964 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "651bac8237ca43fabc87a9897b0d7e51" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:10.226434 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.227088 775 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:10.217322 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:10.197510 5191 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:10.216708 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:10.200469 5340 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:10.227783 663 raft_consensus.cc:2454] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
I20260806 03:44:10.224059 661 rpcz_store.cc:276] 0806 03:44:07.628182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.794219 (+1166037us) service_pool.cc:227] Handling call
0806 03:44:08.837003 (+ 42784us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:08.837227 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.837248 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:08.837661 (+ 413us) log.cc:842] Serialized 26 byte log entry
0806 03:44:09.003188 (+165527us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:44:09.003300 (+ 112us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:09.003380 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:10.222687 (+1219307us) raft_consensus.cc:1671] finished
0806 03:44:10.222707 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:10.222828 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":103497,"threads_started":1,"wal-append.queue_time_us":205499}
I20260806 03:44:10.201725 5318 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.208748 4911 raft_consensus.cc:2781] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:44:10.208940 5361 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.195448 4963 consensus_queue.cc:1035] T b19e6696d8d549f58265db62b449e706 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.019s
I20260806 03:44:10.230962 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.234444 775 raft_consensus.cc:3037] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 10
I20260806 03:44:10.215761 767 raft_consensus.cc:1226] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:10.231276 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:10.235023 668 raft_consensus.cc:1226] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 8. Current term is 9. Ops: []
I20260806 03:44:10.235167 664 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:10.237968 659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 457) took 2609 ms. Trace:
I20260806 03:44:10.198904 5334 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260806 03:44:10.230715 5340 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:10.195030 5362 consensus_queue.cc:1159] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" responder_term: 6 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:44:10.240723 5340 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.238989 5191 raft_consensus.cc:2781] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:44:10.219620 5411 raft_consensus.cc:2781] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:10.199172 5338 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.268133 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:10.268993 616 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:10.238615 659 rpcz_store.cc:276] 0806 03:44:07.628807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.916587 (+1287780us) service_pool.cc:227] Handling call
0806 03:44:08.927410 (+ 10823us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:08.977276 (+ 49866us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:08.977303 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:08.985498 (+ 8195us) log.cc:842] Serialized 26 byte log entry
0806 03:44:09.069565 (+ 84067us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:44:09.069618 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:09.069759 (+ 141us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:10.163472 (+1093713us) spinlock_profiling.cc:243] Waited 5.16 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:44:10.237527 (+ 74055us) raft_consensus.cc:1671] finished
0806 03:44:10.237566 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:10.237886 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34915,"raft.queue_time_us":39675,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":14173,"spinlock_wait_cycles":11860352,"thread_start_us":40634,"threads_started":1,"wal-append.queue_time_us":256346}
I20260806 03:44:10.270536 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:10.244163 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.272946 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.273990 767 raft_consensus.cc:2379] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:10.210158 5428 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:10.248504 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.276296 664 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
I20260806 03:44:10.201973 5373 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.346725 668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2472) took 2745 ms. Trace:
I20260806 03:44:10.281337 768 raft_consensus.cc:2454] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 10.
I20260806 03:44:10.419693 767 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:10.423516 771 raft_consensus.cc:2379] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 5: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:10.305914 727 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:10.290027 776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46314 (request call id 700) took 1209 ms. Trace:
I20260806 03:44:10.425737 770 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
W20260806 03:44:10.366544 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 228051us
W20260806 03:44:10.368140 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 232909us
I20260806 03:44:10.425688 776 rpcz_store.cc:276] 0806 03:44:09.080372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:09.080887 (+ 515us) service_pool.cc:227] Handling call
0806 03:44:10.289775 (+1208888us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540030ff54. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:10.289935 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2019501312}
I20260806 03:44:10.345788 663 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 6.
I20260806 03:44:10.332682 665 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 11.
I20260806 03:44:10.421357 668 rpcz_store.cc:276] 0806 03:44:07.601412 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.790476 (+1189064us) service_pool.cc:227] Handling call
0806 03:44:08.827686 (+ 37210us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:10.235732 (+1408046us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:10.346332 (+110600us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540015c7d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:10.346637 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3235988864}
I20260806 03:44:10.437726 674 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
W20260806 03:44:10.385478 5398 log.cc:924] Time spent T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 1.130s user 0.011s sys 0.003s
I20260806 03:44:10.480036 767 raft_consensus.cc:1226] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
W20260806 03:44:10.310050 5405 log.cc:924] Time spent T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.678s user 0.005s sys 0.011s
I20260806 03:44:10.266103 4911 raft_consensus.cc:686] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [term 10 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:10.466228 615 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:10.288398 5423 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260806 03:44:10.399354 5338 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:10.469902 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:10.289206 5406 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:10.388147 5419 log.cc:924] Time spent T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20260806 03:44:10.303553 5026 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:10.514991 5026 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:10.303552 5206 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.272507 5191 raft_consensus.cc:686] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [term 5 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:10.517292 5191 consensus_queue.cc:237] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.496063 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:10.471697 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 165851us
I20260806 03:44:10.525164 727 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.280124 5411 raft_consensus.cc:686] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:10.366895 5362 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:10.503729 669 raft_consensus.cc:1226] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:10.422397 5428 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:10.305034 5125 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.418735 5253 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:10.329659 5318 consensus_queue.cc:1035] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.172s
I20260806 03:44:10.541499 1663 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:44:10.537027 671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46602 (request call id 456) took 2908 ms. Trace:
I20260806 03:44:10.524686 775 raft_consensus.cc:2454] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 10.
I20260806 03:44:10.480573 5312 raft_consensus.cc:2726] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:10.249799 5361 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:10.536027 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 312558us
I20260806 03:44:10.515558 5026 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:10.552448 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.545195 727 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:10.551682 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.563479 775 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 9.
W20260806 03:44:10.536756 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 70680.7us
I20260806 03:44:10.555657 669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48530 (request call id 2485) took 1727 ms. Trace:
I20260806 03:44:10.567435 669 rpcz_store.cc:276] 0806 03:44:08.827741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.977233 (+149492us) service_pool.cc:227] Handling call
0806 03:44:08.994698 (+ 17465us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:10.533303 (+1538605us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:10.555346 (+ 22043us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54002c0954. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:10.555578 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3469964032}
I20260806 03:44:10.589884 5312 consensus_queue.cc:1035] T 736f0260dd3a4e77a6115e55fa0c4269 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:44:10.542804 671 rpcz_store.cc:276] 0806 03:44:07.628516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.897459 (+1268943us) service_pool.cc:227] Handling call
0806 03:44:08.897840 (+ 381us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:08.977223 (+ 79383us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:44:08.977255 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:09.016666 (+ 39411us) log.cc:842] Serialized 26 byte log entry
0806 03:44:09.299586 (+282920us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:44:09.299643 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:09.299739 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:10.219165 (+919426us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b44000232cc. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f321323bf11 00007f3213309a95 00007f321330a4b2 00007f3215fb2511 00007f3215fade1a 00007f3215fa1bee 00007f3215fa0f36 00007f3215fa0b58 00007f3215faa551 00007f3215faa509 00007f3215faa499
0806 03:44:10.522140 (+302975us) raft_consensus.cc:1671] finished
0806 03:44:10.522163 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:10.536938 (+ 14775us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60551,"raft.queue_time_us":61249,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":11154,"spinlock_wait_cycles":3431296,"thread_start_us":252510,"threads_started":1,"wal-append.queue_time_us":151068}
I20260806 03:44:10.611974 773 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 03:44:10.613719 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 1.60946e+06us
I20260806 03:44:10.538847 5428 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.614939 501 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:10.550693 5243 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:10.504334 5356 consensus_queue.cc:1035] T f37d457494b64bb48d2f06013a9d26c0 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.091s
I20260806 03:44:10.578635 5401 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:10.624451 667 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:10.596416 5323 raft_consensus.cc:2726] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 03:44:10.627355 1622 leader_election.cc:334] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.939s (SENT)
I20260806 03:44:10.625483 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:10.634639 1622 leader_election.cc:334] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.189s (SENT)
I20260806 03:44:10.635355 1622 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:44:10.638000 1622 leader_election.cc:334] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.260s (SENT)
I20260806 03:44:10.543413 5198 consensus_queue.cc:1046] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20260806 03:44:10.641886 1622 leader_election.cc:334] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.260s (SENT)
I20260806 03:44:10.349924 5394 consensus_queue.cc:260] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.610519 5206 raft_consensus.cc:482] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.536175 5362 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.646835 1622 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.542047 5125 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.619256 5355 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:10.664314 1622 connection.cc:232] RPC call timeout handler was delayed by -1.5401668741833419s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26930000000000004s for a response
W20260806 03:44:10.673018 1622 leader_election.cc:334] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 2.776s (SENT)
I20260806 03:44:10.673467 1622 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:10.674515 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 207441us
I20260806 03:44:10.625401 5360 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.675489 5100 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:44:10.675626 5360 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.555750 5253 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.678817 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:10.679447 769 raft_consensus.cc:1226] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
I20260806 03:44:10.680244 5253 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.682053 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.678313 5360 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.641769 5433 raft_consensus.cc:2781] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:10.690029 5264 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:10.640182 5287 raft_consensus.cc:2726] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:10.531329 5406 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.594581 5333 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.651357 5323 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.668836 5334 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:10.668043 5312 raft_consensus.cc:482] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:44:10.723461 667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46602 (request call id 452) took 3087 ms (client timeout 2230 ms). Trace:
I20260806 03:44:10.692026 5336 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.106s
W20260806 03:44:10.728327 667 rpcz_store.cc:269] 0806 03:44:07.627283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:08.056867 (+429584us) service_pool.cc:227] Handling call
0806 03:44:10.714720 (+2657853us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54000b1154. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:10.714880 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5610641792}
W20260806 03:44:10.656320 4963 consensus_queue.cc:1159] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" responder_term: 8 status { last_received { term: 7 index: 6 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260806 03:44:10.730077 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 43686.5us
I20260806 03:44:10.705840 5326 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.660310 4405 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.718758 5253 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:44:10.750193 1662 raft_consensus.cc:1226] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Rejecting Update request from peer ada5ef6656cb4420baf77a4b4342909e for earlier term 9. Current term is 10. Ops: []
I20260806 03:44:10.753615 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:10.667484 5206 raft_consensus.cc:504] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:10.764660 772 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.617s user 0.006s sys 0.005s
I20260806 03:44:10.643369 4911 consensus_queue.cc:237] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.773303 772 raft_consensus.cc:2454] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
W20260806 03:44:10.773665 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 88524.1us
I20260806 03:44:10.776706 766 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
W20260806 03:44:10.777458 772 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57212 (request call id 2426) took 1468 ms (client timeout 1599 ms). Trace:
W20260806 03:44:10.778124 772 rpcz_store.cc:269] 0806 03:44:09.309491 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:09.310228 (+ 737us) service_pool.cc:227] Handling call
0806 03:44:10.777370 (+1467142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:44:10.779261 501 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:10.729981 5433 raft_consensus.cc:686] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:10.713320 4934 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260806 03:44:10.788455 614 leader_election.cc:302] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:10.758960 5253 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:10.791504 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
W20260806 03:44:10.791976 727 connection.cc:232] RPC call timeout handler was delayed by -1.130703111179173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
I20260806 03:44:10.726935 5371 consensus_queue.cc:1035] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260806 03:44:10.794552 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:10.647184 5411 consensus_queue.cc:237] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.795429 769 raft_consensus.cc:2454] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
W20260806 03:44:10.790088 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 104750us
I20260806 03:44:10.725258 5323 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.772130 5356 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:10.803820 773 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:44:10.774502 5147 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:44:10.795666 498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.002s sys 0.009s
W20260806 03:44:10.806948 498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s user 0.002s sys 0.009s
I20260806 03:44:10.796586 727 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:10.797021 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:10.780267 5100 raft_consensus.cc:482] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:10.812243 501 leader_election.cc:302] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
W20260806 03:44:10.822700 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 631454us
I20260806 03:44:10.821003 727 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.823915 498 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.796216 5326 consensus_queue.cc:1035] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
I20260806 03:44:10.795090 5355 raft_consensus.cc:2726] T 20fcbc41710a42cd9ee5dacc5d9577fb P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:10.794855 5253 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:10.788130 5373 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:10.740038 5312 raft_consensus.cc:504] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.821456 5100 raft_consensus.cc:504] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:10.833020 498 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:10.805915 5308 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:10.862654 5308 raft_consensus.cc:482] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:10.863132 5308 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:10.863571 5362 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:10.864854 5453 log.cc:924] Time spent T c2a414fbc8cc4236a2f89a5bec69c6c2 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.050s user 0.003s sys 0.000s
I20260806 03:44:10.865526 4405 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:10.832350 5336 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:10.866753 5329 raft_consensus.cc:3032] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:10.867373 5329 raft_consensus.cc:729] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:10.806402 5356 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:10.837087 5312 leader_election.cc:288] T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:10.874579 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 71503.7us
W20260806 03:44:10.839228 5431 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.203s user 0.001s sys 0.000s
I20260806 03:44:10.839656 5433 consensus_queue.cc:237] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.840442 5463 raft_consensus.cc:2781] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:10.842276 5406 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:10.845261 4786 consensus_meta.cc:287] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.630s user 0.011s sys 0.000s
I20260806 03:44:10.849226 5360 raft_consensus.cc:2781] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:44:10.851006 5475 raft_consensus.cc:2781] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:10.852919 5125 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.854420 5437 raft_consensus.cc:2726] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260806 03:44:10.856176 5394 raft_consensus.cc:3037] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:10.856525 5323 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.861294 5334 raft_consensus.cc:3032] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:44:10.873872 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:10.838299 5432 log.cc:924] Time spent T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20260806 03:44:10.876920 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:10.894055 1622 leader_election.cc:334] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.526s (SENT)
W20260806 03:44:10.890404 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 422541us
W20260806 03:44:10.932762 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 112003us
I20260806 03:44:10.868489 5329 consensus_queue.cc:260] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:10.896219 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:10.897907 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.863132 5206 leader_election.cc:288] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.866776 5336 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
W20260806 03:44:10.907859 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 74918.3us
W20260806 03:44:10.923833 5354 consensus_meta.cc:287] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.720s user 0.009s sys 0.003s
I20260806 03:44:10.912174 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.934677 1622 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.936170 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:10.939450 767 raft_consensus.cc:1261] T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:44:10.923010 5265 raft_consensus.cc:2781] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:10.945544 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.905123 5475 raft_consensus.cc:482] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:10.874287 5217 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260806 03:44:10.946684 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:10.991206 1657 raft_consensus.cc:2379] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 11: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:44:10.971091 5368 consensus_meta.cc:287] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.696s user 0.006s sys 0.007s
I20260806 03:44:10.947664 498 leader_election.cc:302] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.876143 5356 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:10.904069 5360 raft_consensus.cc:482] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:44:11.002398 5462 log.cc:924] Time spent T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.261s user 0.002s sys 0.000s
I20260806 03:44:11.002619 5360 raft_consensus.cc:3037] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:44:11.002665 5428 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.004942 5428 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:10.920596 5100 leader_election.cc:288] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:10.947259 5333 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 10 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:10.977528 727 leader_election.cc:334] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.526s (SENT)
W20260806 03:44:10.975152 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 81142.7us
W20260806 03:44:11.015615 1622 leader_election.cc:334] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639): Timed out: RequestConsensusVote RPC to 127.30.138.66:41639 timed out after 1.728s (SENT)
I20260806 03:44:11.016117 1622 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.926802 5334 raft_consensus.cc:729] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:10.923233 5463 raft_consensus.cc:686] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:10.978999 1671 raft_consensus.cc:2379] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
W20260806 03:44:11.037197 1622 leader_election.cc:334] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 2.693s (SENT)
W20260806 03:44:11.039755 1622 leader_election.cc:334] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 3.162s (SENT)
I20260806 03:44:11.040830 1622 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.045044 5333 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:11.047564 5333 consensus_queue.cc:1046] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260806 03:44:11.051820 5100 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260806 03:44:10.972373 499 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:10.949638 5336 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.947782 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.057145 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68a9e5190380411684128482e3a899b8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:10.997596 5468 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:44:10.998016 5471 raft_consensus.cc:482] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:11.060719 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 253178us
I20260806 03:44:10.985397 5475 raft_consensus.cc:3037] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:10.972291 5329 raft_consensus.cc:3037] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:10.974581 5265 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:11.065101 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.008124 5147 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:10.978238 5125 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:44:11.010824 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 63279.1us
I20260806 03:44:11.070712 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.011950 31273 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:44:10.899097 5362 consensus_queue.cc:1046] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260806 03:44:11.014925 727 leader_election.cc:302] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:10.940642 5264 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:11.038695 5338 raft_consensus.cc:2726] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:44:10.998456 5457 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:44:11.062763 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 113124us
I20260806 03:44:11.062876 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.064292 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.067411 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.095357 661 raft_consensus.cc:2379] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 10: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:11.068846 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.068270 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:11.096863 669 raft_consensus.cc:2362] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8. Re-sending same reply.
I20260806 03:44:11.036028 5368 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.069362 5265 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:11.091702 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.116593 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:11.117800 564 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 9: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
W20260806 03:44:11.117676 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 46654.6us
I20260806 03:44:11.119019 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:11.079281 5125 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:11.070267 5147 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:11.116830 5318 raft_consensus.cc:2726] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:44:11.092845 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 283772us
I20260806 03:44:11.139025 614 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.139108 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.067461 5191 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:44:11.122875 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 145381us
I20260806 03:44:11.142572 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.144537 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.066007 5471 raft_consensus.cc:504] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:11.151546 5401 consensus_meta.cc:287] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.508s user 0.005s sys 0.006s
I20260806 03:44:11.160137 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:11.092629 5369 raft_consensus.cc:482] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:11.162024 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.142622 498 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.162508 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.163676 498 leader_election.cc:302] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.171156 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.179070 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.179342 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.179814 669 raft_consensus.cc:2379] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:11.079011 5026 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.179237 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.181820 659 raft_consensus.cc:2379] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 11: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:11.121122 5361 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.140432 5125 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:11.093461 5264 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.189283 498 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.159687 5368 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.196906 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:11.198889 498 leader_election.cc:302] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.162900 5451 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:11.202171 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae397e5cb5a14531a20bf02d8c71254d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.181128 4405 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:11.205060 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.208369 667 raft_consensus.cc:1261] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:44:11.184687 5401 raft_consensus.cc:2726] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:11.205916 671 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
I20260806 03:44:11.188819 5198 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260806 03:44:11.220633 767 raft_consensus.cc:1261] T 55d4d892dae44f9b919acae13e8d6642 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:44:11.214188 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.216143 564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37194 (request call id 1477) took 1097 ms. Trace:
I20260806 03:44:11.202169 5451 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:11.246337 664 raft_consensus.cc:2379] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:11.204761 5338 raft_consensus.cc:2781] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:11.244071 775 raft_consensus.cc:1226] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:11.189983 5191 raft_consensus.cc:2726] T 651bac8237ca43fabc87a9897b0d7e51 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:11.244663 498 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:11.093340 5334 consensus_queue.cc:260] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.246129 564 rpcz_store.cc:276] 0806 03:44:10.118746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:10.118909 (+ 163us) service_pool.cc:227] Handling call
0806 03:44:11.215954 (+1097045us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b540031f1d4. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:11.216086 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2275332352}
I20260806 03:44:11.247025 614 leader_election.cc:302] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:11.245262 5198 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:11.248600 5406 consensus_queue.cc:1035] T 1fdb062ea0544f0ab7202b067aaa662e P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:11.249418 5026 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.210839 4911 raft_consensus.cc:2781] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:11.208976 4934 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260806 03:44:11.278329 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa826ec82b0a402580f530e7eb832852" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.202839 5356 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.257107 5451 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:11.190256 5369 raft_consensus.cc:504] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.258064 5338 raft_consensus.cc:482] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:11.122529 5463 consensus_queue.cc:237] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.205438 5206 raft_consensus.cc:2726] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:11.290190 5198 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:44:11.288327 5362 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:11.293864 4911 raft_consensus.cc:482] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:11.334486 5362 raft_consensus.cc:482] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:11.326498 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:11.335155 5362 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:11.335326 548 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:11.325755 5500 consensus_queue.cc:1035] T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260806 03:44:11.338335 548 raft_consensus.cc:1261] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:44:11.340876 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:11.334645 4911 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:11.325542 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.341738 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
W20260806 03:44:11.309044 5340 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 1.068s user 0.014s sys 0.001s
I20260806 03:44:11.319056 767 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:44:11.267488 5308 raft_consensus.cc:504] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.357147 562 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 03:44:11.345316 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 100747us
I20260806 03:44:11.346900 674 raft_consensus.cc:1226] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 03:44:11.341674 5360 raft_consensus.cc:504] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.349347 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:11.325968 5434 raft_consensus.cc:2726] T 1c7b59fcc4bf4cafad47c9aed88c416e P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:11.371676 664 raft_consensus.cc:2362] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8. Re-sending same reply.
I20260806 03:44:11.358815 498 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.359573 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "760b94c8ff89432c8cbbb7ed61be23b1" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:11.360051 561 raft_consensus.cc:1226] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:11.316885 5338 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:11.379887 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.331357 5264 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.383999 498 leader_election.cc:302] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.386391 498 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.386782 768 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 9: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
I20260806 03:44:11.387923 498 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.389271 498 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.348940 614 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:11.378392 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:11.393442 664 raft_consensus.cc:2362] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6. Re-sending same reply.
I20260806 03:44:11.402015 498 leader_election.cc:302] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.404495 5433 raft_consensus.cc:3032] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:44:11.407323 5433 raft_consensus.cc:729] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:11.408433 663 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
W20260806 03:44:11.409799 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 60953.2us
I20260806 03:44:11.318974 5253 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.464133 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:11.402681 5411 raft_consensus.cc:2781] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:11.405479 5373 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:11.357947 5340 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.474521 5264 consensus_queue.cc:1035] T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:44:11.457072 5354 raft_consensus.cc:2726] T 8ba60616917a49ff9bdbfbe90b175c6c P ada5ef6656cb4420baf77a4b4342909e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:44:11.476881 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:11.539309 664 raft_consensus.cc:2379] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:11.476680 5336 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:11.463302 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 63663.6us
I20260806 03:44:11.460860 5434 raft_consensus.cc:2741] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260806 03:44:11.464438 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 53818.5us
I20260806 03:44:11.470820 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.543133 558 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 9: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:11.396905 5478 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:11.564260 5329 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:44:11.483304 5368 raft_consensus.cc:2726] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:44:11.488016 5433 consensus_queue.cc:260] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.566587 5433 raft_consensus.cc:3037] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:11.568037 5336 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.570963 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.575173 668 raft_consensus.cc:1226] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:11.487295 4405 consensus_queue.cc:1035] T ac21875cbf2a4ad6b9eee7859bc361fa P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260806 03:44:11.564685 5340 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.456867 5471 leader_election.cc:288] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.614116 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.488857 5411 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:11.658979 5308 leader_election.cc:288] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.659757 5198 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:11.663079 5511 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260806 03:44:11.667204 558 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:11.668561 558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.672237 5360 leader_election.cc:288] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:11.674420 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 453421us
W20260806 03:44:11.675643 5491 log.cc:924] Time spent T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.376s user 0.005s sys 0.000s
I20260806 03:44:11.677386 5125 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.680035 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:11.691493 561 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:11.662133 5198 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.700208 5253 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.702919 5334 raft_consensus.cc:3037] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:11.696025 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:11.709317 775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46314 (request call id 712) took 1123 ms. Trace:
I20260806 03:44:11.709510 5333 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:11.717301 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.718510 767 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:11.716131 775 rpcz_store.cc:276] 0806 03:44:10.586083 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:10.586279 (+ 196us) service_pool.cc:227] Handling call
0806 03:44:10.586707 (+ 428us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:44:11.263026 (+676319us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:11.709034 (+446008us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b5400320854. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f321631ff51 00007f321631e99e 00007f321b2602e9 00007f3214946684 00007f3214946612 00007f3214946541 00007f32149464bc 00007f3214945772 00007f321474994c 00007f3214748d86
0806 03:44:11.709246 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1491924352}
I20260806 03:44:11.727741 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:11.692728 5147 consensus_meta.cc:287] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.568s user 0.019s sys 0.000s
I20260806 03:44:11.712028 5484 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:11.734555 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.734784 499 leader_election.cc:302] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.735399 561 raft_consensus.cc:3037] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:44:11.711035 5492 log.cc:924] Time spent T 1c7b59fcc4bf4cafad47c9aed88c416e P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.523s user 0.000s sys 0.012s
I20260806 03:44:11.689565 5411 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:11.617270 5356 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.764047 767 raft_consensus.cc:2454] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 10.
I20260806 03:44:11.513736 5451 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.765003 768 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260806 03:44:11.784125 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:11.775683 5368 raft_consensus.cc:2726] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:11.767645 501 leader_election.cc:302] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:11.737159 5287 consensus_queue.cc:1046] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20260806 03:44:11.766608 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 364684us
I20260806 03:44:11.795109 616 leader_election.cc:302] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:11.725997 5333 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.795135 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:11.797165 5487 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:44:11.805052 616 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:11.805392 767 raft_consensus.cc:1261] T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260806 03:44:11.803521 498 leader_election.cc:302] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.737002 5147 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.807194 663 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:44:11.805260 5465 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.008s sys 0.021s
I20260806 03:44:11.827962 498 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.763447 5484 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.835047 498 leader_election.cc:302] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.800205 5451 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.838311 1664 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:11.808076 5253 raft_consensus.cc:2781] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:44:11.849022 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:11.807729 5369 leader_election.cc:288] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.857046 1671 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 03:44:11.793509 5026 consensus_queue.cc:1046] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 03:44:11.855095 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:11.830196 5465 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.009s sys 0.021s
I20260806 03:44:11.833330 5340 raft_consensus.cc:3037] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:11.832847 5499 raft_consensus.cc:2726] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:44:11.861280 498 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:11.866058 561 raft_consensus.cc:2454] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 9.
I20260806 03:44:11.835345 5455 raft_consensus.cc:2781] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:44:11.872620 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.872864 614 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:11.873198 561 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:11.867233 615 leader_election.cc:302] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:11.836673 5487 consensus_queue.cc:1225] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer ada5ef6656cb4420baf77a4b4342909e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:44:11.874689 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:11.878626 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 48811.7us
W20260806 03:44:11.880288 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 124941us
I20260806 03:44:11.877058 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:11.883181 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:11.883771 544 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:44:11.832809 5368 consensus_queue.cc:1035] T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260806 03:44:11.886984 561 raft_consensus.cc:2454] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:11.890273 614 leader_election.cc:302] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters:
W20260806 03:44:11.876624 5510 log.cc:924] Time spent T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.390s user 0.001s sys 0.000s
I20260806 03:44:11.897228 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:11.882011 5475 consensus_meta.cc:287] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.811s user 0.011s sys 0.001s
I20260806 03:44:11.876545 4963 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:11.880484 5369 raft_consensus.cc:2726] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:11.903388 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:11.889302 5512 log.cc:924] Time spent T 93b5917270cd4146bb9b6c0ac0dc8414 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260806 03:44:11.905494 5329 raft_consensus.cc:2781] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:11.919123 5147 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.919289 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
W20260806 03:44:11.920166 5494 log.cc:924] Time spent T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.675s user 0.002s sys 0.005s
I20260806 03:44:11.923429 5026 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260806 03:44:11.927930 5338 consensus_meta.cc:287] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.524s user 0.009s sys 0.000s
W20260806 03:44:11.909744 5513 log.cc:924] Time spent T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
W20260806 03:44:11.910449 5522 log.cc:924] Time spent T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
W20260806 03:44:11.933856 5362 consensus_meta.cc:287] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.598s user 0.008s sys 0.000s
I20260806 03:44:11.934307 5362 raft_consensus.cc:504] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.881209 5125 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:11.913327 5484 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:11.938740 548 raft_consensus.cc:2379] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:11.891115 5499 raft_consensus.cc:2726] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:11.940296 5362 leader_election.cc:288] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.906210 5475 raft_consensus.cc:504] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.888228 5253 raft_consensus.cc:686] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 10 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:11.943547 5475 leader_election.cc:288] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:11.943642 5253 consensus_queue.cc:237] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.944322 5360 raft_consensus.cc:482] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:11.930392 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:11.945303 767 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:11.888654 5455 raft_consensus.cc:686] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [term 10 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:11.944906 5360 raft_consensus.cc:504] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:11.911260 5333 leader_election.cc:288] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:11.931366 5265 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.778s user 0.006s sys 0.006s
W20260806 03:44:11.918356 4911 consensus_meta.cc:287] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.562s user 0.009s sys 0.000s
I20260806 03:44:11.885638 5434 raft_consensus.cc:2781] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election won for term 9
W20260806 03:44:11.981523 547 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.007s sys 0.004s
W20260806 03:44:11.984395 547 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.008s sys 0.004s
I20260806 03:44:12.018607 5026 consensus_queue.cc:1035] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260806 03:44:12.021096 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 153951us
I20260806 03:44:11.951303 5125 raft_consensus.cc:482] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:11.952000 5329 raft_consensus.cc:686] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:12.029042 767 raft_consensus.cc:2454] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:12.032954 1624 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:12.035647 5287 raft_consensus.cc:2781] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:12.036244 5287 raft_consensus.cc:686] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.037173 5287 consensus_queue.cc:237] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.984263 5265 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.042410 5265 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:12.043138 542 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 8: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.043740 5455 consensus_queue.cc:237] T 9d2a8e791d4943d9a81d8d8c75d09783 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.045027 5360 leader_election.cc:288] T 43a6bdb2bc794303966a31eeb84328e9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:12.046893 5329 consensus_queue.cc:237] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:11.952942 5338 raft_consensus.cc:504] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.051030 5340 raft_consensus.cc:2726] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:12.051229 5468 raft_consensus.cc:3032] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:12.051651 5468 raft_consensus.cc:729] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.013882 5406 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 03:44:12.052539 5468 consensus_queue.cc:260] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.053817 5468 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:11.983336 4911 raft_consensus.cc:504] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.060462 4911 leader_election.cc:288] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:12.060727 5338 leader_election.cc:288] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:12.033892 5125 raft_consensus.cc:504] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.061494 562 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.062885 664 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260806 03:44:12.063200 5125 leader_election.cc:288] T fb67a142feaf496f862fbd7f06a925f9 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:12.080463 616 leader_election.cc:302] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:11.990829 5434 raft_consensus.cc:686] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 9 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:44:12.083163 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 49849.7us
I20260806 03:44:12.088060 544 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 12.
W20260806 03:44:12.092135 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 201943us
I20260806 03:44:12.097097 1622 leader_election.cc:302] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.099931 727 leader_election.cc:302] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.101339 1622 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:12.111779 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a414fbc8cc4236a2f89a5bec69c6c2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:12.152110 772 raft_consensus.cc:1226] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260806 03:44:12.154747 727 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.156147 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:12.158313 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:12.158365 775 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 9: Already voted for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in this term.
W20260806 03:44:12.161449 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 47122.3us
I20260806 03:44:12.162261 667 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6 [term 8 LEADER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:12.163333 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:12.164448 4911 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:44:12.164707 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdbf67cdfa6f4f2785238e161fd2c4c8" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:12.090015 5489 consensus_queue.cc:1046] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260806 03:44:12.161799 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd5ae69655db414685b926cfecc43d65" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:12.166260 4911 consensus_queue.cc:1046] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:44:12.167271 4911 raft_consensus.cc:3032] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:12.167613 4911 raft_consensus.cc:729] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:12.168342 4911 consensus_queue.cc:260] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.169459 4911 raft_consensus.cc:3037] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:12.170001 665 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:12.170871 5217 consensus_queue.cc:1046] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 03:44:12.107301 5333 raft_consensus.cc:2726] T 68a9e5190380411684128482e3a899b8 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:12.173022 671 raft_consensus.cc:1226] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 12 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 11. Current term is 12. Ops: []
I20260806 03:44:12.175377 766 raft_consensus.cc:1261] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:44:12.115192 5451 raft_consensus.cc:2726] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:44:12.176342 5434 consensus_queue.cc:237] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.156286 5147 consensus_queue.cc:1035] T c2a414fbc8cc4236a2f89a5bec69c6c2 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260806 03:44:12.149890 5334 raft_consensus.cc:2781] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:12.171116 776 raft_consensus.cc:1261] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 03:44:12.178751 5334 raft_consensus.cc:686] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [term 8 LEADER]: Becoming Leader. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.179800 5334 consensus_queue.cc:237] T d1e65c896c1a47c4a0a7eb7b1036bbba P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.183796 5465 consensus_queue.cc:1046] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 03:44:12.184788 5465 raft_consensus.cc:3032] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:12.185204 5465 raft_consensus.cc:729] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.188241 5411 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.189152 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:12.190469 558 raft_consensus.cc:2379] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.192615 5489 raft_consensus.cc:3032] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:44:12.192842 616 leader_election.cc:302] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:12.193073 5489 raft_consensus.cc:729] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.193666 5198 raft_consensus.cc:2726] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:12.195593 663 raft_consensus.cc:1226] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 6. Current term is 7. Ops: []
W20260806 03:44:12.199381 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 44753.7us
I20260806 03:44:12.158933 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:12.200265 727 leader_election.cc:302] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.112951 5125 raft_consensus.cc:2726] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:44:12.211947 727 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.217005 727 leader_election.cc:302] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.225683 1622 leader_election.cc:302] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.194015 5489 consensus_queue.cc:260] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.225672 5459 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:44:12.227937 5489 raft_consensus.cc:3037] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:44:12.224499 5411 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:12.230181 5305 consensus_queue.cc:1046] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 03:44:12.231231 5305 consensus_queue.cc:1225] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer d8469561a52c4bbc81dce786f1211ee6 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260806 03:44:12.231706 5305 raft_consensus.cc:3032] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:12.232015 5305 raft_consensus.cc:729] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.232781 5479 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:12.232833 5100 consensus_queue.cc:1046] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 6, Time since last communication: 0.004s
I20260806 03:44:12.223210 5360 raft_consensus.cc:3037] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:12.237180 5125 raft_consensus.cc:2726] T ae397e5cb5a14531a20bf02d8c71254d P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:12.186069 5465 consensus_queue.cc:260] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.241173 5100 raft_consensus.cc:3032] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 LEADER]: Stepping down as leader of term 11
I20260806 03:44:12.256218 5100 raft_consensus.cc:729] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.232843 5305 consensus_queue.cc:260] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.271045 5318 raft_consensus.cc:2726] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:44:12.283528 616 leader_election.cc:302] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.302795 766 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
W20260806 03:44:12.316455 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 88144.4us
I20260806 03:44:12.296590 5468 consensus_queue.cc:1046] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.011s
I20260806 03:44:12.298137 5455 raft_consensus.cc:2726] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 03:44:12.319933 5549 log.cc:924] Time spent T 329fbbb627464f478b8d2fd7a28851cc P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20260806 03:44:12.322404 5100 consensus_queue.cc:260] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.323849 5100 raft_consensus.cc:3037] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:44:12.325263 5371 raft_consensus.cc:3032] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:12.325856 5371 raft_consensus.cc:729] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.327044 5336 consensus_queue.cc:1046] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:44:12.332278 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 471036us
I20260806 03:44:12.332890 5433 raft_consensus.cc:2726] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:12.335631 5394 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:12.343289 5308 raft_consensus.cc:3032] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:44:12.343880 5308 raft_consensus.cc:729] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:12.344671 5308 consensus_queue.cc:260] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.346829 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:12.347482 773 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.347767 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
I20260806 03:44:12.350984 768 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:44:12.319976 5468 raft_consensus.cc:3032] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:12.355213 5468 raft_consensus.cc:729] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1914b652c9fb49b694d71a5f5f5752ee, State: Running, Role: LEADER
I20260806 03:44:12.356334 5468 consensus_queue.cc:260] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.357856 5468 raft_consensus.cc:3037] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.358523 614 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.358795 5465 raft_consensus.cc:3037] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.301200 5198 consensus_queue.cc:1035] T 20df53c445204494999245a03bc56a91 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260806 03:44:12.359776 614 leader_election.cc:302] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:12.366914 766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:12.301759 5329 consensus_queue.cc:1035] T ae397e5cb5a14531a20bf02d8c71254d P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
I20260806 03:44:12.367614 766 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
W20260806 03:44:12.371310 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 154393us
W20260806 03:44:12.363623 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 574533us
I20260806 03:44:12.354007 5356 raft_consensus.cc:3032] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:12.373462 614 leader_election.cc:302] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.373798 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8119c361164a62a5ab1232752156bb" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:12.300546 5305 raft_consensus.cc:3037] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:12.374668 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a4c59f28ea45f6998c3f7f0b7dda53" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:12.378463 564 raft_consensus.cc:1226] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:12.379442 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
W20260806 03:44:12.653846 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 280327us
I20260806 03:44:12.658823 614 leader_election.cc:302] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.605093 727 leader_election.cc:302] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.660578 614 leader_election.cc:302] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 1914b652c9fb49b694d71a5f5f5752ee, ada5ef6656cb4420baf77a4b4342909e
W20260806 03:44:12.608143 1622 leader_election.cc:334] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.917s (SENT)
I20260806 03:44:12.362514 5540 raft_consensus.cc:3037] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.606343 546 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:12.605389 558 raft_consensus.cc:1226] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 5. Current term is 6. Ops: []
I20260806 03:44:12.391104 5356 raft_consensus.cc:729] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:12.327077 5371 consensus_queue.cc:260] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:12.608623 615 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 232403us
I20260806 03:44:12.602722 668 raft_consensus.cc:1226] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [term 9 LEADER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 8. Current term is 9. Ops: []
I20260806 03:44:12.613823 5318 raft_consensus.cc:482] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:12.614120 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:12.617106 5125 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:12.617697 498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 271244us
I20260806 03:44:12.617851 5394 consensus_queue.cc:1035] T 760b94c8ff89432c8cbbb7ed61be23b1 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260806 03:44:12.618635 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 240936us
I20260806 03:44:12.651604 5455 raft_consensus.cc:2726] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:44:12.625125 5541 raft_consensus.cc:482] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.633716 5217 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.639448 5459 raft_consensus.cc:2726] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 03:44:12.642598 5542 raft_consensus.cc:2781] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 12 FOLLOWER]: Leader election won for term 12
I20260806 03:44:12.346299 5308 raft_consensus.cc:3037] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:44:12.649923 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:12.650182 1670 raft_consensus.cc:2454] T ba8119c361164a62a5ab1232752156bb P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:12.382002 773 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:12.721616 562 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d8469561a52c4bbc81dce786f1211ee6 in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.660212 499 leader_election.cc:302] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee
I20260806 03:44:12.685659 5564 raft_consensus.cc:2726] T aa826ec82b0a402580f530e7eb832852 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:44:12.662703 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 54604.9us
W20260806 03:44:12.678253 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 73253.7us
I20260806 03:44:12.723382 727 leader_election.cc:302] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.602712 5360 raft_consensus.cc:2726] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:12.692065 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.699676 5334 raft_consensus.cc:482] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:12.700310 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.624622 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee"
I20260806 03:44:12.690696 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.661665 5198 raft_consensus.cc:2726] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:12.706517 501 leader_election.cc:302] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:12.685264 5125 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.673839 5451 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.708447 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.709211 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.709900 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93b5917270cd4146bb9b6c0ac0dc8414" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.664245 5329 raft_consensus.cc:2726] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:12.682204 5499 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.745586 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2018f00b21544838b05e402c218b85ea" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.739953 727 leader_election.cc:302] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters: ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:12.747125 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20df53c445204494999245a03bc56a91" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.685851 5336 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.686044 5489 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:12.725037 499 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 64994.1us
I20260806 03:44:12.726936 544 raft_consensus.cc:2354] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:12.743769 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc95e3152e264afab52ba5a7cd5458a7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.693158 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864f4306a7964687b312ccf31fe6797a" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.777210 1670 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 03:44:12.731698 564 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.716661 5217 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.695896 5318 raft_consensus.cc:504] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.699781 4405 consensus_queue.cc:1046] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 03:44:12.750833 5334 raft_consensus.cc:504] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.671734 5356 consensus_queue.cc:260] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.801723 5265 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.010s
I20260806 03:44:12.811581 5356 raft_consensus.cc:3037] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.811815 5265 consensus_queue.cc:1225] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Peer 1914b652c9fb49b694d71a5f5f5752ee log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260806 03:44:12.755990 5459 raft_consensus.cc:2781] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:44:12.814966 5459 raft_consensus.cc:686] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:12.816316 5459 consensus_queue.cc:237] T 8b0f8d511d8e4dce9973f791ed3bb702 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.722359 5437 consensus_queue.cc:1035] T fc95e3152e264afab52ba5a7cd5458a7 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260806 03:44:12.796761 5489 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.766535 401 catalog_manager.cc:5495] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
W20260806 03:44:12.768577 5559 log.cc:924] Time spent T ce36e9a0e95c408ca319bfb41b336e31 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.140s user 0.003s sys 0.000s
I20260806 03:44:12.710976 5541 raft_consensus.cc:504] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.772648 5125 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:12.785576 5451 raft_consensus.cc:504] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.773221 5499 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.752722 501 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e; no voters:
I20260806 03:44:12.777380 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcd22bae8db4028ba3d0e499cd18e13" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.799968 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bea9147cd344c6d8e771c5d9968f2e7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.829073 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.801700 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c5936af1dcf436eb5f06e8fc0765e83" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.830055 544 raft_consensus.cc:2379] T 43a6bdb2bc794303966a31eeb84328e9 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 6: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.752727 5340 raft_consensus.cc:2781] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:44:12.777927 563 raft_consensus.cc:2454] T 864f4306a7964687b312ccf31fe6797a P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:12.831352 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:12.787436 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b8bbd25b8646eb93f739aafb4d00f8" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.832425 542 raft_consensus.cc:2454] T 90b8bbd25b8646eb93f739aafb4d00f8 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 8.
I20260806 03:44:12.833443 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "497d006930504e8184f1e5e512ade026" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.788892 1662 raft_consensus.cc:2396] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260806 03:44:12.724545 4963 raft_consensus.cc:2726] T e8a4c59f28ea45f6998c3f7f0b7dda53 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:12.803488 1671 raft_consensus.cc:2396] T b635b355749342d0ad7857884a590b8b P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 03:44:12.806679 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.841557 563 raft_consensus.cc:2354] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7.
I20260806 03:44:12.806902 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "736f0260dd3a4e77a6115e55fa0c4269" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:12.796988 5336 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.810678 561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37194 (request call id 1507) took 1235 ms. Trace:
I20260806 03:44:12.835382 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:12.824571 403 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:51378 (request call id 180) took 12406 ms (client timeout 15000 ms). Trace:
W20260806 03:44:12.827834 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 75214.7us
I20260806 03:44:12.829315 547 raft_consensus.cc:2454] T 2bea9147cd344c6d8e771c5d9968f2e7 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:12.842758 561 rpcz_store.cc:276] 0806 03:44:11.575559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:11.887676 (+312117us) service_pool.cc:227] Handling call
0806 03:44:11.887981 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:44:12.336812 (+448831us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:44:12.336841 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:44:12.337362 (+ 521us) log.cc:842] Serialized 26 byte log entry
0806 03:44:12.652989 (+315627us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:44:12.653143 (+ 154us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:44:12.653413 (+ 270us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:44:12.810444 (+157031us) raft_consensus.cc:1671] finished
0806 03:44:12.810464 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:44:12.810617 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":448416,"raft.queue_time_us":448562,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":2816,"thread_start_us":313316,"threads_started":1,"wal-append.queue_time_us":470121}
I20260806 03:44:12.845345 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ba60616917a49ff9bdbfbe90b175c6c" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.841452 548 raft_consensus.cc:2362] T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 5. Re-sending same reply.
I20260806 03:44:12.865916 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.825323 401 catalog_manager.cc:5495] T 2018f00b21544838b05e402c218b85ea P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 4 to 5, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:12.868295 546 raft_consensus.cc:2379] T ce36e9a0e95c408ca319bfb41b336e31 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 8: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.872320 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.872773 771 raft_consensus.cc:1226] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Rejecting Update request from peer d8469561a52c4bbc81dce786f1211ee6 for earlier term 6. Current term is 7. Ops: []
I20260806 03:44:12.873531 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:12.874363 542 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.875435 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.883414 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9369dafb3f342e7a15d44086a377b55" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.827504 5355 raft_consensus.cc:482] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.818327 5362 raft_consensus.cc:3032] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:44:12.891485 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.823174 5541 leader_election.cc:288] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:12.823558 5318 leader_election.cc:288] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:12.840677 5340 raft_consensus.cc:482] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:12.825671 5451 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:12.820497 5489 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:12.845049 403 rpcz_store.cc:269] 0806 03:44:00.418245 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:00.418451 (+ 206us) service_pool.cc:227] Handling call
0806 03:44:12.824475 (+12406024us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:44:08.778337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:44:08.778988 (+ 651us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:44:08.779042 (+ 54us) write_op.cc:435] Acquired schema lock
0806 03:44:08.779059 (+ 17us) tablet.cc:660] Decoding operations
0806 03:44:08.780610 (+ 1551us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:44:08.780742 (+ 132us) write_op.cc:641] Partition lock acquired for write op
0806 03:44:08.780768 (+ 26us) tablet.cc:683] Acquiring locks for 17 operations
0806 03:44:08.782098 (+ 1330us) tablet.cc:699] Row locks acquired
0806 03:44:08.782146 (+ 48us) write_op.cc:260] PREPARE: finished
0806 03:44:08.782316 (+ 170us) write_op.cc:270] Start()
0806 03:44:08.782516 (+ 200us) write_op.cc:276] Timestamp: P: 1785987848782248 usec, L: 0
0806 03:44:08.782539 (+ 23us) op_driver.cc:331] REPLICATION: starting
0806 03:44:08.783307 (+ 768us) log.cc:842] Serialized 6665 byte log entry
0806 03:44:10.710093 (+1926786us) op_driver.cc:444] REPLICATION: finished
0806 03:44:11.058861 (+348768us) write_op.cc:301] APPLY: starting
0806 03:44:11.063216 (+ 4355us) tablet.cc:1365] starting BulkCheckPresence
0806 03:44:11.136824 (+ 73608us) tablet.cc:1368] finished BulkCheckPresence
0806 03:44:11.136963 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:44:11.713964 (+577001us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:44:11.714239 (+ 275us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0806 03:44:11.714280 (+ 41us) write_op.cc:312] APPLY: finished
0806 03:44:11.714650 (+ 370us) log.cc:842] Serialized 119 byte log entry
0806 03:44:11.885352 (+170702us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:44:11.906564 (+ 21212us) write_op.cc:454] Released schema lock
0806 03:44:11.950726 (+ 44162us) write_op.cc:341] FINISH: Updating metrics
0806 03:44:12.013458 (+ 62732us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:44:12.017253 (+ 3795us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19052,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":290571,"apply.run_cpu_time_us":27665,"apply.run_wall_time_us":983775,"num_ops":17,"prepare.queue_time_us":322977,"prepare.run_cpu_time_us":51252,"prepare.run_wall_time_us":142590,"raft.queue_time_us":321704,"raft.run_cpu_time_us":14784,"raft.run_wall_time_us":157842,"replication_time_us":1913554,"spinlock_wait_cycles":7424,"thread_start_us":907073,"threads_started":5,"wal-append.queue_time_us":377062,"wal-append.run_cpu_time_us":48796,"wal-append.run_wall_time_us":2680420}]]}
W20260806 03:44:12.877017 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 137182us
I20260806 03:44:12.893805 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.827016 5334 leader_election.cc:288] T 1fdb062ea0544f0ab7202b067aaa662e P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:12.900350 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:12.904721 564 raft_consensus.cc:1226] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [term 10 LEADER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 9. Current term is 10. Ops: []
I20260806 03:44:12.907078 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.899569 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:12.899969 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:12.894769 5265 raft_consensus.cc:482] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:12.878543 5437 raft_consensus.cc:2781] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:44:12.866343 5360 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
W20260806 03:44:12.865545 5100 consensus_meta.cc:287] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.541s user 0.013s sys 0.000s
I20260806 03:44:12.921896 5340 raft_consensus.cc:3037] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:12.867945 5499 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:12.884430 5564 raft_consensus.cc:2781] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:44:12.925065 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e12cd5292fc847be9a0e9d515b7faf96" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:12.927206 768 raft_consensus.cc:1261] T 5502493e599c44abbc15f805d5298dbc P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 03:44:12.943729 772 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 7.
I20260806 03:44:12.945492 727 leader_election.cc:302] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.972929 727 leader_election.cc:302] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:12.892180 561 raft_consensus.cc:1226] T 30edb33481c941c08c9a8430b4d0bc54 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Rejecting Update request from peer 0374a84d5e9f4e3c8d8287e2382379c8 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260806 03:44:12.946549 548 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 9: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:12.944381 770 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 11.
I20260806 03:44:12.894856 4963 raft_consensus.cc:482] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:12.949476 564 raft_consensus.cc:1226] T 329fbbb627464f478b8d2fd7a28851cc P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Rejecting Update request from peer 1914b652c9fb49b694d71a5f5f5752ee for earlier term 7. Current term is 8. Ops: []
I20260806 03:44:12.905452 5362 raft_consensus.cc:729] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:12.984421 401 catalog_manager.cc:5495] T 8ba60616917a49ff9bdbfbe90b175c6c P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:12.955821 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502493e599c44abbc15f805d5298dbc" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:12.875038 5542 raft_consensus.cc:686] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [term 12 LEADER]: Becoming Leader. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
I20260806 03:44:12.927713 5487 raft_consensus.cc:482] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:12.926553 5217 leader_election.cc:288] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:12.979427 727 leader_election.cc:302] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:12.903025 5355 raft_consensus.cc:504] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:12.970564 5437 raft_consensus.cc:686] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [term 7 LEADER]: Becoming Leader. State: Replica: ada5ef6656cb4420baf77a4b4342909e, State: Running, Role: LEADER
I20260806 03:44:12.981316 5125 raft_consensus.cc:3037] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:13.019259 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.026324 727 leader_election.cc:302] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:12.995023 5564 raft_consensus.cc:686] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Leader. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:13.036229 563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.036273 548 raft_consensus.cc:2396] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260806 03:44:12.961593 5265 raft_consensus.cc:504] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.040417 401 catalog_manager.cc:5495] T 864f4306a7964687b312ccf31fe6797a P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 6 to 7, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 7 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:12.975345 5360 raft_consensus.cc:504] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:12.998766 5484 consensus_queue.cc:1035] T 5502493e599c44abbc15f805d5298dbc P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.043s
I20260806 03:44:12.993988 4963 raft_consensus.cc:504] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:13.060696 1622 leader_election.cc:334] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.937s (SENT)
I20260806 03:44:13.023916 5579 raft_consensus.cc:2726] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:13.063661 1622 leader_election.cc:302] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:13.080629 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecf1b55970984ed5ad958ef3c517c6b5" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:13.081717 548 raft_consensus.cc:2379] T ecf1b55970984ed5ad958ef3c517c6b5 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee in current term 7: Already voted for candidate ada5ef6656cb4420baf77a4b4342909e in this term.
I20260806 03:44:13.068706 546 raft_consensus.cc:2454] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:13.037637 5576 raft_consensus.cc:2741] T 14cd85313f7a4ff7a17b8200a6317da4 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260806 03:44:13.018179 5487 raft_consensus.cc:504] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.101325 563 raft_consensus.cc:2396] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260806 03:44:13.042138 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 319804us
W20260806 03:44:13.101312 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 203154us
I20260806 03:44:13.107893 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.108779 770 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 7.
I20260806 03:44:13.126160 616 leader_election.cc:302] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:13.125725 564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
I20260806 03:44:13.129110 564 raft_consensus.cc:2396] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1914b652c9fb49b694d71a5f5f5752ee for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260806 03:44:13.116801 401 catalog_manager.cc:5495] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:13.137492 616 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6; no voters:
I20260806 03:44:13.139083 401 catalog_manager.cc:5495] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:13.140872 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.146873 561 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:13.147840 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25056084174744caba1a79c3eb002598" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.142561 564 raft_consensus.cc:1261] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:44:13.098330 5540 raft_consensus.cc:2726] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:13.143844 770 raft_consensus.cc:1261] T 9d2a8e791d4943d9a81d8d8c75d09783 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 10 FOLLOWER]: Refusing update from remote peer d8469561a52c4bbc81dce786f1211ee6: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260806 03:44:13.149878 767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.155478 767 raft_consensus.cc:2454] T 44d06aad0812478bb2270a2c80917306 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:13.114086 5336 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:13.163168 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.282208 768 raft_consensus.cc:2454] T e1fbe5a0e51744568ce535d54093a890 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 11.
I20260806 03:44:13.099606 5318 consensus_queue.cc:1046] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260806 03:44:13.285230 401 catalog_manager.cc:5495] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:13.079692 5362 consensus_queue.cc:260] T 497d006930504e8184f1e5e512ade026 P ada5ef6656cb4420baf77a4b4342909e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.289641 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.150630 5198 raft_consensus.cc:2781] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:13.326992 5198 raft_consensus.cc:482] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:13.327587 5198 raft_consensus.cc:3037] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:44:13.327828 5308 consensus_meta.cc:287] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.604s user 0.011s sys 0.000s
W20260806 03:44:13.279227 5573 log.cc:924] Time spent T 25056084174744caba1a79c3eb002598 P d8469561a52c4bbc81dce786f1211ee6: Append to log took a long time: real 0.328s user 0.001s sys 0.001s
I20260806 03:44:13.313416 5265 leader_election.cc:288] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.315286 5576 raft_consensus.cc:3032] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:44:13.086215 5542 consensus_queue.cc:237] T 921ac79ba4d94c039b073809e4a17f11 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:13.290230 5547 log.cc:924] Time spent T 5bc28108ed7945c88b5b7e3560f525e5 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20260806 03:44:13.319983 5355 leader_election.cc:288] T 3bcd22a4b7694c71b676f8939a238cd0 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.325852 5487 leader_election.cc:288] T d1e65c896c1a47c4a0a7eb7b1036bbba P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:13.326285 5478 raft_consensus.cc:482] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:13.311600 4963 leader_election.cc:288] T f37d457494b64bb48d2f06013a9d26c0 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:13.331679 401 catalog_manager.cc:5495] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:13.319150 5360 leader_election.cc:288] T b19e6696d8d549f58265db62b449e706 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:13.344501 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.66:41639, user_credentials={real_user=slave}} blocked reactor thread for 304604us
I20260806 03:44:13.349967 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.292074 5540 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260806 03:44:13.371325 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 231201us
I20260806 03:44:13.337073 5576 raft_consensus.cc:729] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0374a84d5e9f4e3c8d8287e2382379c8, State: Running, Role: LEADER
I20260806 03:44:13.373505 561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.381481 561 raft_consensus.cc:2454] T edcb8ef1a79d4601b243ec9ce6e6acb7 P ada5ef6656cb4420baf77a4b4342909e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 10.
W20260806 03:44:13.389672 727 leader_election.cc:334] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.520s (SENT)
I20260806 03:44:13.390192 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.390345 727 leader_election.cc:302] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:13.132745 5437 consensus_queue.cc:237] T 14cd85313f7a4ff7a17b8200a6317da4 P ada5ef6656cb4420baf77a4b4342909e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.407964 674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc28108ed7945c88b5b7e3560f525e5" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.408689 674 raft_consensus.cc:3037] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:44:13.409094 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac21875cbf2a4ad6b9eee7859bc361fa" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.409744 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "604c561db3f3435da75d2536381f6b96" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:13.409982 727 connection.cc:232] RPC call timeout handler was delayed by -1.2261167610995471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20260806 03:44:13.410473 727 connection.cc:232] RPC call timeout handler was delayed by -1.1703167611267418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
W20260806 03:44:13.410776 727 connection.cc:232] RPC call timeout handler was delayed by -1.1694167610257864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260806 03:44:13.411113 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:13.411653 727 leader_election.cc:334] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761): Timed out: RequestConsensusVote RPC to 127.30.138.65:45761 timed out after 1.655s (SENT)
I20260806 03:44:13.412714 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "329fbbb627464f478b8d2fd7a28851cc" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:13.410496 614 leader_election.cc:302] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:13.413192 548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:13.412096 727 leader_election.cc:302] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:13.410715 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb67a142feaf496f862fbd7f06a925f9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:13.416062 664 raft_consensus.cc:1261] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:44:13.417258 664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.371258 5540 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:13.412463 667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721de14efa984f9ebf40c801a835c6ac" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:13.420570 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce36e9a0e95c408ca319bfb41b336e31" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.413808 548 raft_consensus.cc:2454] T 721de14efa984f9ebf40c801a835c6ac P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 6.
I20260806 03:44:13.421514 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30edb33481c941c08c9a8430b4d0bc54" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.428838 659 raft_consensus.cc:2379] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
W20260806 03:44:13.425724 727 connection.cc:232] RPC call timeout handler was delayed by -1.0776167609728873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1887s for a response
I20260806 03:44:13.429896 727 leader_election.cc:302] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:13.431639 663 raft_consensus.cc:2454] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:13.432945 669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55827d5ca304ba3a2fb74f4a6e5d1c4" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:44:13.435218 773 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.001s sys 0.025s
W20260806 03:44:13.441125 773 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s user 0.002s sys 0.025s
I20260806 03:44:13.441075 663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b635b355749342d0ad7857884a590b8b" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
W20260806 03:44:13.442508 616 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 289703us
I20260806 03:44:13.431919 674 raft_consensus.cc:2454] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:13.443848 669 raft_consensus.cc:2379] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 8: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:13.442209 663 raft_consensus.cc:2379] T b635b355749342d0ad7857884a590b8b P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:13.445048 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d4d892dae44f9b919acae13e8d6642" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.446210 668 raft_consensus.cc:2379] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 5: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:13.451566 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43a6bdb2bc794303966a31eeb84328e9" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.452557 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e65c896c1a47c4a0a7eb7b1036bbba" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.451496 661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ec448ca686a4641a5af84944f63b7e2" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.453717 661 raft_consensus.cc:2379] T 7ec448ca686a4641a5af84944f63b7e2 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 9: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:13.457758 401 catalog_manager.cc:5495] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 4 to 6, leader changed from ada5ef6656cb4420baf77a4b4342909e (127.30.138.65) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } }
I20260806 03:44:13.366554 547 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0488766430fc45ddb1edb22dd107266d"
dest_uuid: "ada5ef6656cb4420baf77a4b4342909e"
mode: GRACEFUL
new_leader_uuid: "0374a84d5e9f4e3c8d8287e2382379c8"
from {username='slave'} at 127.0.0.1:43080
I20260806 03:44:13.465370 547 raft_consensus.cc:595] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Received request to transfer leadership to TS 0374a84d5e9f4e3c8d8287e2382379c8
I20260806 03:44:13.465934 547 raft_consensus.cc:603] T 0488766430fc45ddb1edb22dd107266d P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:44:13.468722 665 raft_consensus.cc:1261] T ac21875cbf2a4ad6b9eee7859bc361fa P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260806 03:44:13.471390 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 61762.4us
I20260806 03:44:13.472522 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.286859 5564 consensus_queue.cc:237] T 3bcd22a4b7694c71b676f8939a238cd0 P 0374a84d5e9f4e3c8d8287e2382379c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.432799 5540 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P d8469561a52c4bbc81dce786f1211ee6 [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:44:13.432694 5579 raft_consensus.cc:2726] T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:13.435286 5484 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:13.468466 5593 log.cc:924] Time spent T c9369dafb3f342e7a15d44086a377b55 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 03:44:13.443127 5459 raft_consensus.cc:2726] T e8b626bb795440e18f02bbdcbd10f9ce P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:13.365969 5478 raft_consensus.cc:504] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.427476 5541 consensus_queue.cc:1046] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260806 03:44:13.446028 5577 raft_consensus.cc:2741] T ac21875cbf2a4ad6b9eee7859bc361fa P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 03:44:13.513039 665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3b590a95fa411a91b9a3818bf7d29f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6"
I20260806 03:44:13.522599 665 raft_consensus.cc:2379] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e in current term 7: Already voted for candidate d8469561a52c4bbc81dce786f1211ee6 in this term.
I20260806 03:44:13.515480 401 catalog_manager.cc:5495] T 651bac8237ca43fabc87a9897b0d7e51 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:13.470789 5621 raft_consensus.cc:482] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:13.499711 5329 raft_consensus.cc:2726] T 30edb33481c941c08c9a8430b4d0bc54 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:44:13.524866 401 catalog_manager.cc:5495] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 reported cstate change: term changed from 2 to 6, leader changed from 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67) to d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66). New cstate: current_term: 6 leader_uuid: "d8469561a52c4bbc81dce786f1211ee6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } health_report { overall_health: HEALTHY } } }
I20260806 03:44:13.500250 5265 raft_consensus.cc:482] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:13.511910 5579 raft_consensus.cc:2726] T 2018f00b21544838b05e402c218b85ea P 0374a84d5e9f4e3c8d8287e2382379c8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:44:13.574051 5603 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
I20260806 03:44:13.575176 5478 leader_election.cc:288] T 7a9afce5efa4491797ebed2aab67532f P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:13.513620 5484 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:13.575922 5604 log.cc:924] Time spent T fc95e3152e264afab52ba5a7cd5458a7 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 03:44:13.577662 5484 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.529728 5499 raft_consensus.cc:482] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.578439 5499 raft_consensus.cc:504] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.534415 5621 raft_consensus.cc:504] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.480466 5576 consensus_queue.cc:260] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.580160 5265 raft_consensus.cc:504] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:13.593122 5265 leader_election.cc:288] T e8b626bb795440e18f02bbdcbd10f9ce P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:13.595507 5125 consensus_meta.cc:287] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8: Time spent flushing consensus metadata: real 0.558s user 0.008s sys 0.001s
I20260806 03:44:13.596107 5125 raft_consensus.cc:2726] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:13.596998 769 raft_consensus.cc:2454] T d1e65c896c1a47c4a0a7eb7b1036bbba P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:13.596145 5617 raft_consensus.cc:3032] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:44:13.605966 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.580392 5499 leader_election.cc:288] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.607028 5371 raft_consensus.cc:3037] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:13.613938 5468 raft_consensus.cc:482] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.614033 5147 raft_consensus.cc:482] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.614732 5465 raft_consensus.cc:482] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.614681 5468 raft_consensus.cc:504] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.615316 5465 raft_consensus.cc:504] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.616794 5468 leader_election.cc:288] T 921ac79ba4d94c039b073809e4a17f11 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:13.617935 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 557242us
I20260806 03:44:13.618391 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8b626bb795440e18f02bbdcbd10f9ce" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.619385 5563 raft_consensus.cc:2726] T c9369dafb3f342e7a15d44086a377b55 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:44:13.620817 1622 connection.cc:232] RPC call timeout handler was delayed by -1.0000248099677265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
I20260806 03:44:13.622007 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:13.624678 1624 leader_election.cc:302] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
W20260806 03:44:13.556931 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 53986.4us
I20260806 03:44:13.629208 664 raft_consensus.cc:2454] T 14cd85313f7a4ff7a17b8200a6317da4 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 7.
I20260806 03:44:13.615355 5489 raft_consensus.cc:482] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0374a84d5e9f4e3c8d8287e2382379c8)
I20260806 03:44:13.614749 5147 raft_consensus.cc:504] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.634652 5584 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.634749 5489 raft_consensus.cc:504] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.635241 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14cd85313f7a4ff7a17b8200a6317da4" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:13.585790 5356 consensus_meta.cc:287] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6: Time spent flushing consensus metadata: real 0.774s user 0.008s sys 0.004s
I20260806 03:44:13.636299 5147 leader_election.cc:288] T 9d2a8e791d4943d9a81d8d8c75d09783 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.636842 5489 leader_election.cc:288] T 736f0260dd3a4e77a6115e55fa0c4269 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.637123 5465 leader_election.cc:288] T 44d06aad0812478bb2270a2c80917306 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:13.588179 5621 leader_election.cc:288] T 68a9e5190380411684128482e3a899b8 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
W20260806 03:44:13.637916 5605 log.cc:924] Time spent T 736f0260dd3a4e77a6115e55fa0c4269 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.063s user 0.001s sys 0.001s
I20260806 03:44:13.639573 1662 pending_rounds.cc:76] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Aborting all ops after (but not including) 8
I20260806 03:44:13.639891 1662 pending_rounds.cc:97] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260806 03:44:13.640177 1662 raft_consensus.cc:2866] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:44:13.640684 665 raft_consensus.cc:1261] T fc95e3152e264afab52ba5a7cd5458a7 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Refusing update from remote peer ada5ef6656cb4420baf77a4b4342909e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:44:13.640869 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edcb8ef1a79d4601b243ec9ce6e6acb7" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:13.629738 668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
W20260806 03:44:13.643635 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 213820us
I20260806 03:44:13.644479 727 leader_election.cc:302] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:13.645774 1670 raft_consensus.cc:2379] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0374a84d5e9f4e3c8d8287e2382379c8 in current term 10: Already voted for candidate 1914b652c9fb49b694d71a5f5f5752ee in this term.
I20260806 03:44:13.645766 671 raft_consensus.cc:2454] T 43a6bdb2bc794303966a31eeb84328e9 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 6.
I20260806 03:44:13.648499 1658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:13.657163 775 raft_consensus.cc:1261] T dd5ae69655db414685b926cfecc43d65 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 8 FOLLOWER]: Refusing update from remote peer 1914b652c9fb49b694d71a5f5f5752ee: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260806 03:44:13.694041 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 53699.8us
W20260806 03:44:13.694285 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 68633.1us
I20260806 03:44:13.601709 5617 raft_consensus.cc:729] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d8469561a52c4bbc81dce786f1211ee6, State: Running, Role: LEADER
W20260806 03:44:13.584398 5340 consensus_meta.cc:287] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e: Time spent flushing consensus metadata: real 0.598s user 0.007s sys 0.002s
I20260806 03:44:13.693195 614 leader_election.cc:302] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8469561a52c4bbc81dce786f1211ee6; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:13.823374 1657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f37d457494b64bb48d2f06013a9d26c0" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:14.087142 1624 leader_election.cc:302] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8, 1914b652c9fb49b694d71a5f5f5752ee; no voters:
I20260806 03:44:14.093658 671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d8469561a52c4bbc81dce786f1211ee6" is_pre_election: true
I20260806 03:44:14.091825 544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b19e6696d8d549f58265db62b449e706" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
W20260806 03:44:13.821048 1671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60258 (request call id 975) took 1402 ms (client timeout 1613 ms). Trace:
I20260806 03:44:14.092545 498 leader_election.cc:302] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 1914b652c9fb49b694d71a5f5f5752ee, d8469561a52c4bbc81dce786f1211ee6
I20260806 03:44:13.625579 5576 raft_consensus.cc:3037] T 30edb33481c941c08c9a8430b4d0bc54 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:44:13.986791 499 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.30.138.68:33823 timed out after 1.613s (SENT) [suppressed 28 similar messages]
I20260806 03:44:14.108623 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9afce5efa4491797ebed2aab67532f" candidate_uuid: "ada5ef6656cb4420baf77a4b4342909e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
W20260806 03:44:14.109594 727 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 465163us
W20260806 03:44:14.109933 614 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 416843us
I20260806 03:44:13.633291 5287 raft_consensus.cc:2781] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:44:14.116852 727 leader_election.cc:302] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0374a84d5e9f4e3c8d8287e2382379c8; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:14.111343 1671 rpcz_store.cc:269] 0806 03:44:12.418993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:12.613855 (+194862us) service_pool.cc:227] Handling call
0806 03:44:13.820739 (+1206884us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54004f2754. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:13.820948 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":278860672}
I20260806 03:44:14.106114 671 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ada5ef6656cb4420baf77a4b4342909e in term 8.
I20260806 03:44:14.121482 1658 raft_consensus.cc:2454] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0374a84d5e9f4e3c8d8287e2382379c8 in term 7.
I20260806 03:44:14.121728 562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fdb062ea0544f0ab7202b067aaa662e" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:14.122731 562 raft_consensus.cc:2454] T 1fdb062ea0544f0ab7202b067aaa662e P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 8.
I20260806 03:44:14.123991 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921ac79ba4d94c039b073809e4a17f11" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:14.124902 546 raft_consensus.cc:2454] T 921ac79ba4d94c039b073809e4a17f11 P ada5ef6656cb4420baf77a4b4342909e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1914b652c9fb49b694d71a5f5f5752ee in term 12.
I20260806 03:44:14.075502 5305 raft_consensus.cc:482] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:14.104578 5466 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.943s user 0.006s sys 0.015s
W20260806 03:44:13.749516 5606 log.cc:924] Time spent T 55d4d892dae44f9b919acae13e8d6642 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.233s user 0.000s sys 0.001s
I20260806 03:44:13.819696 5451 raft_consensus.cc:482] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:14.129961 498 leader_election.cc:302] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ada5ef6656cb4420baf77a4b4342909e; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:14.126780 5475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.798s user 0.014s sys 0.012s
I20260806 03:44:14.086414 5631 raft_consensus.cc:482] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:13.704984 5125 raft_consensus.cc:2726] T 20df53c445204494999245a03bc56a91 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:44:13.819470 5579 raft_consensus.cc:482] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:14.141639 5308 raft_consensus.cc:482] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.908876 5433 raft_consensus.cc:482] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:13.819077 5618 raft_consensus.cc:482] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:13.590154 5198 raft_consensus.cc:504] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.113071 5329 raft_consensus.cc:2726] T 5bc28108ed7945c88b5b7e3560f525e5 P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:13.652894 5622 consensus_queue.cc:1035] T 604c561db3f3435da75d2536381f6b96 P d8469561a52c4bbc81dce786f1211ee6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.485s
I20260806 03:44:14.108947 5582 raft_consensus.cc:2781] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260806 03:44:14.137598 5466 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.983s user 0.007s sys 0.016s
I20260806 03:44:13.716053 5217 consensus_queue.cc:1035] T ecf1b55970984ed5ad958ef3c517c6b5 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.431s
I20260806 03:44:13.635274 5584 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.122262 5287 raft_consensus.cc:482] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Starting leader election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:14.140523 5499 raft_consensus.cc:482] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:44:14.159578 5564 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.000s sys 0.036s
W20260806 03:44:14.169565 4911 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.023s sys 0.027s
I20260806 03:44:14.204598 5125 raft_consensus.cc:482] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 1914b652c9fb49b694d71a5f5f5752ee)
I20260806 03:44:14.199364 5459 raft_consensus.cc:2726] T 5dcd22bae8db4028ba3d0e499cd18e13 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:44:13.875113 5608 raft_consensus.cc:482] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:14.111095 5340 raft_consensus.cc:504] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.161856 4963 raft_consensus.cc:2726] T 604c561db3f3435da75d2536381f6b96 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 03:44:14.170455 5475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s user 0.014s sys 0.012s
I20260806 03:44:14.144829 5305 raft_consensus.cc:504] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.161914 5451 raft_consensus.cc:504] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:13.648335 5147 consensus_queue.cc:1035] T 90b8bbd25b8646eb93f739aafb4d00f8 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.509s
W20260806 03:44:14.242449 5564 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.000s sys 0.037s
I20260806 03:44:14.230830 5582 raft_consensus.cc:482] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:44:14.241991 5287 raft_consensus.cc:3037] T b19e6696d8d549f58265db62b449e706 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:44:14.198675 5579 raft_consensus.cc:504] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.194780 5631 raft_consensus.cc:504] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.262133 5583 consensus_queue.cc:1035] T dd5ae69655db414685b926cfecc43d65 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.096s
I20260806 03:44:14.163758 5617 consensus_queue.cc:260] T ce36e9a0e95c408ca319bfb41b336e31 P d8469561a52c4bbc81dce786f1211ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.262977 5600 raft_consensus.cc:482] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:14.226137 5618 raft_consensus.cc:504] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.207619 5433 raft_consensus.cc:504] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.299564 5305 leader_election.cc:288] T 329fbbb627464f478b8d2fd7a28851cc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:14.299983 5451 leader_election.cc:288] T c2a414fbc8cc4236a2f89a5bec69c6c2 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:14.300467 5433 leader_election.cc:288] T 2018f00b21544838b05e402c218b85ea P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.301026 5584 leader_election.cc:288] T ce3b590a95fa411a91b9a3818bf7d29f P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:14.302997 5618 leader_election.cc:288] T 55d4d892dae44f9b919acae13e8d6642 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.303846 5600 raft_consensus.cc:504] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:14.250084 4911 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.023s sys 0.027s
I20260806 03:44:14.303856 5579 leader_election.cc:288] T 93b5917270cd4146bb9b6c0ac0dc8414 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:14.308018 5600 leader_election.cc:288] T 760b94c8ff89432c8cbbb7ed61be23b1 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:14.204788 5308 raft_consensus.cc:504] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
W20260806 03:44:14.276072 5371 consensus_meta.cc:287] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee: Time spent flushing consensus metadata: real 0.669s user 0.007s sys 0.006s
I20260806 03:44:14.262318 5394 raft_consensus.cc:2726] T fb67a142feaf496f862fbd7f06a925f9 P ada5ef6656cb4420baf77a4b4342909e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:14.247952 5499 raft_consensus.cc:504] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
W20260806 03:44:14.296728 5630 log.cc:924] Time spent T 90b8bbd25b8646eb93f739aafb4d00f8 P 0374a84d5e9f4e3c8d8287e2382379c8: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260806 03:44:14.322696 5499 leader_election.cc:288] T 497d006930504e8184f1e5e512ade026 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.314571 5479 raft_consensus.cc:482] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:14.327250 5479 raft_consensus.cc:504] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.287617 5582 raft_consensus.cc:3037] T e1fbe5a0e51744568ce535d54093a890 P 1914b652c9fb49b694d71a5f5f5752ee [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:44:14.259907 5608 raft_consensus.cc:504] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } }
W20260806 03:44:14.335446 5640 log.cc:924] Time spent T 5bc28108ed7945c88b5b7e3560f525e5 P ada5ef6656cb4420baf77a4b4342909e: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260806 03:44:14.356415 5608 leader_election.cc:288] T 651bac8237ca43fabc87a9897b0d7e51 P 0374a84d5e9f4e3c8d8287e2382379c8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
I20260806 03:44:14.356402 5631 leader_election.cc:288] T 20df53c445204494999245a03bc56a91 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
W20260806 03:44:14.201316 728 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.68:33823, user_credentials={real_user=slave}} blocked reactor thread for 96167.9us
W20260806 03:44:14.396806 501 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 310681us
W20260806 03:44:14.292414 1622 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.65:45761, user_credentials={real_user=slave}} blocked reactor thread for 134040us
I20260806 03:44:14.237063 667 raft_consensus.cc:2354] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for earlier term 7. Current term is 8.
I20260806 03:44:14.409713 1622 leader_election.cc:302] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: 0374a84d5e9f4e3c8d8287e2382379c8, ada5ef6656cb4420baf77a4b4342909e
I20260806 03:44:14.412429 1622 leader_election.cc:302] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1914b652c9fb49b694d71a5f5f5752ee; no voters: ada5ef6656cb4420baf77a4b4342909e, d8469561a52c4bbc81dce786f1211ee6
W20260806 03:44:14.290278 1624 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.30.138.67:43937, user_credentials={real_user=slave}} blocked reactor thread for 156479us
W20260806 03:44:14.290079 1662 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.000s sys 0.002s
W20260806 03:44:14.420513 1662 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s user 0.001s sys 0.002s
I20260806 03:44:14.252457 5125 raft_consensus.cc:504] T 3d8ac3ebccff4054953475f839218e45 P 0374a84d5e9f4e3c8d8287e2382379c8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.424806 5455 raft_consensus.cc:482] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:44:14.425297 5198 leader_election.cc:288] T 721de14efa984f9ebf40c801a835c6ac P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 8 election: Requested vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.426844 5371 raft_consensus.cc:2726] T edcb8ef1a79d4601b243ec9ce6e6acb7 P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 03:44:14.425907 5455 raft_consensus.cc:504] T ce3b590a95fa411a91b9a3818bf7d29f P d8469561a52c4bbc81dce786f1211ee6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.427983 5371 raft_consensus.cc:2726] T 25056084174744caba1a79c3eb002598 P 1914b652c9fb49b694d71a5f5f5752ee [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:44:14.428398 5338 raft_consensus.cc:482] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:14.429587 5308 leader_election.cc:288] T dd5ae69655db414685b926cfecc43d65 P d8469561a52c4bbc81dce786f1211ee6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:14.429438 5338 raft_consensus.cc:504] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } }
I20260806 03:44:14.464147 546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bcd22a4b7694c71b676f8939a238cd0" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:14.465402 546 raft_consensus.cc:2454] T 3bcd22a4b7694c71b676f8939a238cd0 P ada5ef6656cb4420baf77a4b4342909e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 6.
I20260806 03:44:14.466295 547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0f8d511d8e4dce9973f791ed3bb702" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ada5ef6656cb4420baf77a4b4342909e" is_pre_election: true
I20260806 03:44:14.467808 547 raft_consensus.cc:2454] T 8b0f8d511d8e4dce9973f791ed3bb702 P ada5ef6656cb4420baf77a4b4342909e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8469561a52c4bbc81dce786f1211ee6 in term 8.
I20260806 03:44:14.462815 5338 leader_election.cc:288] T e12cd5292fc847be9a0e9d515b7faf96 P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.471804 5338 raft_consensus.cc:482] T 2c5936af1dcf436eb5f06e8fc0765e83 P ada5ef6656cb4420baf77a4b4342909e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
I20260806 03:44:14.484572 5479 leader_election.cc:288] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), d8469561a52c4bbc81dce786f1211ee6 (127.30.138.66:41639)
W20260806 03:44:14.430555 5437 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.005s sys 0.013s
I20260806 03:44:14.430871 1664 raft_consensus.cc:2396] T f55827d5ca304ba3a2fb74f4a6e5d1c4 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ada5ef6656cb4420baf77a4b4342909e for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260806 03:44:14.431782 5305 raft_consensus.cc:482] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d8469561a52c4bbc81dce786f1211ee6)
W20260806 03:44:14.486091 5437 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s user 0.006s sys 0.015s
I20260806 03:44:14.486961 5305 raft_consensus.cc:504] T fdbf67cdfa6f4f2785238e161fd2c4c8 P 1914b652c9fb49b694d71a5f5f5752ee [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.488481 5489 raft_consensus.cc:482] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ada5ef6656cb4420baf77a4b4342909e)
I20260806 03:44:14.489682 5489 raft_consensus.cc:504] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ada5ef6656cb4420baf77a4b4342909e" member_type: VOTER last_known_addr { host: "127.30.138.65" port: 45761 } } peers { permanent_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" member_type: VOTER last_known_addr { host: "127.30.138.67" port: 43937 } } peers { permanent_uuid: "1914b652c9fb49b694d71a5f5f5752ee" member_type: VOTER last_known_addr { host: "127.30.138.68" port: 33823 } attrs { promote: false } }
I20260806 03:44:14.490538 563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58044 (request call id 493) took 1465 ms. Trace:
I20260806 03:44:14.491091 563 rpcz_store.cc:276] 0806 03:44:13.025171 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:44:13.025319 (+ 148us) service_pool.cc:227] Handling call
0806 03:44:14.489883 (+1464564us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b540030e654. stack: 00007f32133c5033 00007f32133c4f30 00007f3212d23534 00007f321ca2f8b4 00007f321ca2f849 00007f32162a4471 00007f32163232ae 00007f321b260c27 00007f3214948c04 00007f3214948b92 00007f3214948ac1 00007f3214948a3c 00007f3214947cf2 00007f321474994c 00007f3214748d86 00007f321474872f
0806 03:44:14.490209 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":137415552}
I20260806 03:44:14.496388 5489 leader_election.cc:288] T 5502493e599c44abbc15f805d5298dbc P 1914b652c9fb49b694d71a5f5f5752ee [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ada5ef6656cb4420baf77a4b4342909e (127.30.138.65:45761), 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937)
I20260806 03:44:14.433571 5340 leader_election.cc:288] T ba8119c361164a62a5ab1232752156bb P ada5ef6656cb4420baf77a4b4342909e [CANDIDATE]: Term 7 election: Requested vote from peers 0374a84d5e9f4e3c8d8287e2382379c8 (127.30.138.67:43937), 1914b652c9fb49b694d71a5f5f5752ee (127.30.138.68:33823)
I20260806 03:44:14.430181 5371 consensus_queue.cc:1046] T 604c561db3f3435da75d2536381f6b96 P 1914b652c9fb49b694d71a5f5f5752ee [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d8469561a52c4bbc81dce786f1211ee6" member_type: VOTER last_known_addr { host: "127.30.138.66" port: 41639 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260806 03:44:14.438757 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1fbe5a0e51744568ce535d54093a890" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "1914b652c9fb49b694d71a5f5f5752ee" is_pre_election: true
I20260806 03:44:14.442572 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d2a8e791d4943d9a81d8d8c75d09783" candidate_uuid: "1914b652c9fb49b694d71a5f5f5752ee" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "0374a84d5e9f4e3c8d8287e2382379c8" is_pre_election: true
I20260806 03:44:14.444247 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44d06aad0812478bb2270a2c80917306" candidate_uuid: "d8469561a52c4bbc81dce786f1211ee6" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "